acceptodds
Under review as a conference paper at ICLR 2027

SteerProbe: Learning to Bypass Safety Steering in Vision–Language Models

Abstract

Activation steering offers an inference-time defense for vision–language models (VLMs) by modifying intermediate representations without updating backbone parameters. However, protection on benchmark inputs may not persist across alternative expressions of the same harmful request. We investigate this gap using fixed textual, visual, and joint reformulations designed to preserve the underlying intent, and find that these changes can bypass representative steering defenses. A complementary local analysis provides a sufficient condition under which a reformulation can cross a surrogate safety margin despite any admissible change in the local steering correction. We then introduce SteerProbe, an output-only black-box attack that learns to select effective reformulations for unseen requests from a shared calibration budget. Across three VLM backbones, two benchmarks, and three steering defenses, SteerProbe increases Harmful Rate in all defended settings using 500 total calibration queries per endpoint and benchmark, raising the average from 7.43% to 18.36%. These findings highlight that robustness on original benchmark inputs is insufficient to characterize the safety of steering defenses and motivate reformulation robustness as an important evaluation dimension. They further motivate steering mechanisms that preserve safety across intent-preserving multimodal variations while maintaining benign utility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Activation Steering, Jailbreak Attacks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mRmeOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the robustness of steering defenses in VLMs and identities reformulations can substantially increase harmful response rates. Based on this, this paper proposes SteerProbe, an output-only black-box attack that learns to select effective reformulations for unseen requests. Experiments across three VLM backbones, two multimodal jailbreak benchmarks, and three steering defenses show that SteerProbe increases the average harmful. Overall, this paper identity a weakness of current steer-based defenses and provides an interesting direction for future research.

Critical strengths
  • The research problem holds practical significance. This paper focuses on the safety robustness of "Activation Steering" when faced with semantically preserved variations in multimodal inputs. This perspective facilitates a more comprehensive understanding of the actual safety boundaries of "Steering Defense."
  • The attack scenario is highly practical. SteerProbe employs an "output-only" black-box threat model, requiring no access to the target model’s internal parameters, gradients, or hidden states.
  • The methodology is clearly structured and accounts for a limited query budget. The authors decompose the process into distinct modules utilizing lightweight logistic regression to facilitate feedback transfer.
  • The experimental evaluation is thorough. The study assesses performance across three different backbones, three distinct defenses, and two multimodal attack benchmarks. The paper also analyzes the impact of several key factors, and possible defenses.
Critical weaknesses
  1. The safety evaluation metrics are not enough. This paper mainly uses Llama Guard 3-8B to determine the harmfulness of the model output, and uses Harmful Rate (HR) as the core indicator of attack effectiveness. However, some answers that contain sensitive content but do not actually satisfy malicious requests may affect the final evaluation results. It is recommended that the author add attack success rate evaluation based on a general-purpose LLM Judge (such as GPT-4o), taking into account both the original request and the generated reply, to further verify the reliability of the current experimental conclusions.

  2. The design motivation of Logistic Regression as a selector is not sufficient. This paper uses a lightweight Regularized Logistic Regression Model to predict the attack effects of different Reformulations. However, this model is essentially a linear classifier and may struggle to fully capture the complex nonlinear relationships between input features and different rewriting operations. The authors should further explain the reasons for choosing linear models over other predictive models. Furthermore, could using a simple MLP or other lightweight nonlinear model improve the selection of candidate rewrites? It is recommended to supplement relevant comparative experiments to verify the rationality of the current model design.

  3. The analysis of the impact of different Reformulation types is still insufficient. This article mainly considers predefined text rewriting, image transformation and the combination of the two, but how different rewriting categories affect the robustness of Steering Defense is still lacking a more detailed analysis. For example, are there significant performance differences between Direct Grounding, Phrase Injection, Reference Resolution, and Instruction Framing in text rewriting? Are these differences consistent across different Steering Defenses? Furthermore, current candidate rewrites are mainly based on fixed templates and conventional image processing operations, and their coverage is relatively limited. The authors could further discuss the basis for the selection of existing rewriting operations, and whether it is possible to further improve the effectiveness of the attack and enhance the generality of the method when extending the method to more semantically preserving text or visual changes.

  4. There is a lack of sufficient analysis of the training and calibration costs of the method. This paper highlights that SteerProbe requires only one target model query during the testing phase, but the training and calibration overhead of the method is still not fully discussed. Specifically, SteerProbe needs to collect target model feedback through multiple rounds of Active Calibration and repeatedly update the Logistic Regression Selector while performing uncertainty estimation and candidate selection. However, the paper has not yet reported the actual computational cost and running time of these steps. The authors are recommended to supplement the analysis of Training/Calibration Time and overall computational overhead to further evaluate the actual efficiency and scalability of this method.

Questions for authors
  1. In addition to the Harmful Rate (HR) based on Llama Guard 3-8B, have the authors considered using other LLM-based judges (such as GPT-4o) for evaluation assessing the actual attack success rate by analyzing both the original malicious requests and the model responses?
  2. What were the primary considerations behind choosing Regularized Logistic Regression over non-linear models like a simple MLP? Could more complex models yield further improvements in attack effectiveness given the same calibration budget?
  3. Are there significant differences in how various types of Text Reformulation (e.g., Direct Grounding, Phrase Injection, and Reference Resolution) affect different Steering Defenses? Furthermore, given that the current method relies heavily on predefined text templates and image processing operations, have the authors considered extending the approach to include more semantic-preserving reformulation techniques?
4 comments
AuthorsOpenReview ↗

Response to Reviewer mRme (Part: 1/2)

Thanks for your careful review. We sincerely hope the following clarifications can address your concerns. All changes in the revised PDF are highlighted in red.

W1+Q1. The safety evaluation metrics are not enough.

Re: Following your suggestion, we further employ DeepSeek-V4-Flash as an independent multimodal judge. Specifically, the judge receives the original image–text input together with the complete response generated by the target VLM and determines whether the response provides substantive harmful information that follows and fulfills the original malicious intent. We use a binary Yes/No judgment and count only affirmative responses as successful attacks. We report the resulting ASR in the following Table. The results are consistent with Table 1 in the paper.

We include the details and results in the Appendix. D1 in the revised paper.

Table R1. ASR (%) evaluated by DeepSeek-V4-Flash.

ModelDefenseMM-SafetyBench OriginalMM-SafetyBench +SteerProbeHADES OriginalHADES +SteerProbe
Qwen2-VL-7BNone16.0722.2249.7044.64
L2S0.6015.2819.6426.79
NullSteer0.6012.301.7910.71
CMRM_data0.0010.525.3623.21
Qwen3-VL-8BNone0.601.791.7911.31
L2S0.602.980.0014.29
NullSteer0.008.135.3632.14
CMRM_data0.003.570.007.14
LLaVA-1.5-13BNone22.8229.9633.9342.86
L2S0.0013.100.002.98
NullSteer0.0011.115.9514.29
CMRM_data21.0329.7613.9932.14

W2+Q2. The design motivation of Logistic Regression as a selector is not sufficient.

Re: We choose regularized logistic regression for its data efficiency, low computational cost, and tractable uncertainty estimation under limited calibration feedback. Our 508-dimensional feature representation captures request-specific information and reformulation characteristics, enabling input-conditioned candidate selection. Moreover, logistic regression supports efficient iterative fitting and Laplace-based uncertainty estimation, making it suitable for active calibration.

Following your suggestion, we further compare logistic regression with a one-hidden-layer MLP (32 ReLU units), using identical features, 500 calibration observations, and test requests. Logistic regression achieves a 1.79-percentage-point higher HR while requiring approximately 627× less fitting time. These results support its effectiveness and computational efficiency under our limited calibration budget. We include the details and results in the Appendix. D2 in the revised paper.

Table R2. Comparison between the regularized logistic regression selector and a lightweight nonlinear MLP on Qwen2-VL-7B / MM-SafetyBench.

SelectorHR (%)Fitting time (s)
L2-regularized Logistic Regression (Ours)25.600.114
Regularized MLP (32 hidden units)23.8171.677
AuthorsOpenReview ↗

Response to Reviewer mRme (Part: 2/2)

W3+Q3. The analysis of the impact of different Reformulation types is still insufficient.

Re: Following your suggestion, we add a finer-grained analysis of reformulation families and discuss the scope of the current candidate library. We include the details and results in the Appendix. D4 in the revised paper.

1. Effects of different reformulation types. We group the textual candidates into four families: Direct Grounding, Phrase Injection, Reference Resolution, and Instruction Framing. Table R3 reports, for each defense, the proportion of successful final attacks whose selected textual candidate belongs to each family. Thus, the entries describe the composition of successful attacks rather than the standalone HR of each family.

Table R3. Distribution of successful final attacks across textual reformulation families (%).

Reformulation familyL2SNullSteerCMRM_data
Direct Grounding31.2521.8225.81
Phrase Injection37.5045.4522.58
Reference Resolution18.7518.1829.03
Instruction Framing12.5014.5522.58

2. Effect of candidate-library size. We further vary the number of textual candidates while keeping the image-transformation set, calibration budget, selection procedure, and evaluation requests fixed. Table R4 shows that the relationship between library size and HR is not monotonic. These findings indicate that a larger candidate pool can increase coverage, but under a fixed calibration budget it can also make effective candidates harder to identify reliably.

We also agree that broader semantic coverage is an important direction for future work. Future extensions could explore more diverse and semantically preserving reformulations to further expand the coverage of potential vulnerabilities. Our current work primarily aims to identify and systematically expose the limitations of existing safety steering defenses under multimodal reformulations, rather than exhaustively exploring the entire space of possible transformations.

Table R4. HR (%) under different textual candidate-library sizes.

DefenseK = 6K = 12K = 18K = 24
L2S17.2611.3119.0517.26
NullSteer33.9332.7432.7427.38
CMRM_data6.558.9318.4510.12

W4. There is a lack of sufficient analysis of the training and calibration costs of the method.

Re: Following your suggestion, we provide additional measurements to quantify the computational overhead of active calibration and selector fitting. Specifically, the active candidate acquisition process takes 73.74 seconds across 27 rounds, averaging 2.73 seconds per round. After collecting the 500 calibration observations, fitting the final regularized logistic regression selector takes only 0.114 seconds. These results demonstrate that the local acquisition and selector-fitting computations are lightweight. The main calibration cost comes from collecting target-model feedback, while the selector itself can be fitted efficiently. We will include these measurements in the revised manuscript to provide a clearer account of SteerProbe's computational overhead. We include the details and results in the Appendix. D3 in the revised paper.

Table R5. Computational overhead of active calibration and selector fitting.

Computational componentMeasured time
Active candidate acquisition (27 rounds)73.74 s
Average acquisition time per round2.73 s
Final logistic regression fitting0.114 s

Thank you again for your time and feedback. We sincerely hope these clarifications help address your concerns.

Thank you for your detailed response and additional experimental results. At present, my major concerns are solved through these additional experiments and clarifications. The added analysis strengthens the paper's reliability and quality. I will incerese my rating from 3 to 4.

Dear Reviewer mRme,

Thank you very much for your positive feedback and for raising the score to clear acceptance! It encourages us a lot! We sincerely appreciate your recognition of our work's contributions, and your constructive feedback helps us improve the quality of our work. Thanks!

Best regards,

The Authors

Reviewer_risqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper presents an attack framework for VLM safety. It learns to select multimodal reformulations, which transforms the format of queries but retains their harmful intent. The results show that the proposed framework bypasses the safety guard of VLMs.

Critical strengths
  1. The experiments cover 3 backbones and 3 defences, with consistent improvement of attack success rates.

  2. The reformulation-augmented defense experiments also provide interesting perspective on how to solve the studied vulnerability.

Critical weaknesses
  1. The safety evaluation is response-only. It is also important to show that responses indeed follow the multimodal prompts, otherwise the safety evaluation itself may be hacked.

  2. In Appendix D.3, hyperparameters are selected by “held-out” attack performance. A separate validation split should be clarified to exclude test-set tuning.

  3. The theoretical analysis is based on reachability assumptions and a surrogate safety margin. I am not sure if they are general enough.

Questions for authors

Please refer to the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer risq (Part: 1/2)

Thanks for your careful review. We sincerely hope the following clarifications can address your concerns. All changes in the revised PDF are highlighted in red.

W1. The safety evaluation is response-only. It is also important to show that responses indeed follow the multimodal prompts, otherwise the safety evaluation itself may be hacked.

Re: Response-only harmfulness evaluation is a commonly adopted practice in jailbreak research [R1,R2], and we retain it for consistency with existing evaluation protocols. Nevertheless, we agree that harmfulness alone does not guarantee that a response follows the original multimodal request.

Following your suggestion, we further employ DeepSeek-V4-Flash as an independent multimodal judge. Specifically, the judge receives the original image–text input together with the complete response generated by the target VLM and determines whether the response provides substantive harmful information that follows and fulfills the original malicious intent. We use a binary Yes/No judgment and count only affirmative responses as successful attacks. We report the resulting ASR in the following Table. The results are consistent with Table 1 in the paper.

We include the details and results in the Appendix. D1 in the revised paper.

Table R1. ASR (%) evaluated by DeepSeek-V4-Flash.

ModelDefenseMM-SafetyBench OriginalMM-SafetyBench +SteerProbeHADES OriginalHADES +SteerProbe
Qwen2-VL-7BNone16.0722.2249.7044.64
L2S0.6015.2819.6426.79
NullSteer0.6012.301.7910.71
CMRM_data0.0010.525.3623.21
Qwen3-VL-8BNone0.601.791.7911.31
L2S0.602.980.0014.29
NullSteer0.008.135.3632.14
CMRM_data0.003.570.007.14
LLaVA-1.5-13BNone22.8229.9633.9342.86
L2S0.0013.100.002.98
NullSteer0.0011.115.9514.29
CMRM_data21.0329.7613.9932.14

[R1] Hu, Xuhao, et al. "Vlsbench: Unveiling visual leakage in multimodal safety." ACL, 2025

[R2] Zhang, Shenyi, et al. "MMAligner: Safeguarding Multimodal Large Language Models through Representation Calibration." arXiv preprint arXiv:2608.05909 (2026).

W2. In Appendix D.3, hyperparameters are selected by “held-out” attack performance. A separate validation split should be clarified to exclude test-set tuning.

Re: We select hyperparameters once on MM-SafetyBench and directly transfer the resulting configuration to HADES and all other experimental settings without further tuning.

  1. Hyperparameter selection. We use Qwen2-VL-7B on MM-SafetyBench as the representative development setting to determine the active batch size and uncertainty weight. The selected values are a batch size of 16 and (β=1.25\beta=1.25).

  2. Cross-benchmark transfer with fixed hyperparameters. Importantly, these hyperparameters are kept unchanged when evaluating HADES, which uses a different benchmark and source attack construction (VisCRA), as well as other VLM backbones and steering defenses. No HADES test results are used to select or adjust these parameters. The results in Table 1 demonstrate that SteerProbe remains effective across these different configurations using the same hyperparameter settings, suggesting that the selected configuration is not specific to MM-SafetyBench.

AuthorsOpenReview ↗

Response to Reviewer risq (Part: 2/2)

W3. The theoretical analysis is based on reachability assumptions and a surrogate safety margin. I am not sure if they are general enough.

Re: Our theoretical analysis uses a local abstraction to characterize when intent-preserving reformulations can exceed the compensation capacity of steering defenses. The assumptions are intended to capture general characteristics of a broad class of representation-based steering mechanisms.

  1. Reformulation reachability. We introduce this assumption to connect a theoretically adverse representation direction to an actual input reformulation. Specifically, the proof first identifies a representation displacement along which the surrogate safety margin decreases most rapidly within a local subspace. Since such a displacement is not necessarily realizable through an actual input transformation, the reachability assumption ensures that an intent-preserving reformulation can approximately realize it. This abstraction does not prescribe specific transformation operations and can accommodate textual, visual, and joint multimodal reformulations.

  2. Surrogate safety margin. We introduce this assumption to characterize the compensation capacity of steering mechanisms. The correction-change subspace (SDS_D) captures admissible correction directions, while (BDB_D) bounds their magnitude. This provides an architecture-independent abstraction of constrained steering interventions. For example, CMRM in our setting uses a fixed correction, yielding (sD=0s_D=0), while input-dependent and projection-constrained steering mechanisms can be characterized by their respective admissible correction changes, provided that the local boundedness condition holds.


Thank you again for your time and feedback. We sincerely hope these clarifications help address your concerns.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer risq,

Thank you for the detailed questions. To make the discussion easier, we would like to briefly summarize the main additions in our rebuttal.

  1. Following your suggestion, we have introduced an independent multimodal evaluation (W1).
  2. We clarify that the parameters are selected once on MM-SafetyBench and kept fixed across HADES, different backbones, and steering defenses without further tuning (W2)
  3. We have expanded the theoretical discussion to explain the roles of reformulation reachability and the surrogate safety margin, and how these assumptions capture a broad class of representation-based steering mechanisms (W3).

As the discussion deadline approaches, we would be happy to clarify any remaining questions. If your concerns have been resolved, we would appreciate your reconsideration of the rating. Thanks again!

Best Regards,

The Authors

Reviewer_KFgHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author investigates robustness of activation-based safety steering in VLM, and introduces STEERPROBE, an output-only black-box attack that learns a reusable selection rule from a separate calibration set. The method consists of (i) a shared pool of textual and visual reformulations; (ii) an input-conditioned selector that transfers feedback across requests using request text, candidate text, and operation identifiers; and (iii) active calibration that allocates the limited query budget to promising and informative can. The effective results are reported on the multiple datasets.

Critical strengths

1、The paper considers the limitations of existing safety steering methods: robustness on original benchmark inputs is insufficient to characterize the safety of steering defenses and motivate reformulation robustness as an important evaluation dimension. 2、Comprehensive experiments are conducted across three VLM backbones, two benchmarks, and the analysis covers multiple aspects like calibration, selection. 3、The motivation of method is clear and well motivated, the attack setting is practical.

Critical weaknesses

1、The limited novelty. The proposed method combines several existing component which are relatively standard components, and their combination does not introduce a fundamentally new attack optimization or learning mechanism. 2、Theroretical analysis. More analysises are needed to explain the insights into the effectiveness of the proposed method. e.g. How identified failure conditions correspond to the reformulations. More insights should be provided. 3、Comparison methods are insfuccient, and the competitiveness against existing black-box jailbreak attacks should be further validated by more extra comaprisons. 4、The generalization to unseen attack templates, different safety distributions, or other multimodal tasks remains unclear.

Questions for authors

1、Can the author provide more comparison against the existing query-efficient black-box attack methods under equivalent query budgets? 2、The performance gains or which benefit most to the final results, more discussion and analysis could be added. library, selector or calibration. 3、Can the auhtor conduct more evidence investigation and provide more theoretical analysis informs the design of SteerProbe?

4 comments
AuthorsOpenReview ↗

Response to Reviewer KFgH (Part: 1/3)

Thanks for your careful review. We sincerely hope the following clarifications can address your concerns. All changes in the revised PDF are highlighted in red.

W1. The limited novelty.

Re. Our novelty primarily lies in identifying and systematically investigating an overlooked vulnerability of existing safety steering defenses, rather than introducing fundamentally new optimization components. We would like to clarify our contributions from two perspectives:

1. Identifying an overlooked vulnerability. Existing VLM jailbreak studies mainly investigate attacks against the underlying models, whereas we examine whether safety steering defenses remain effective when the same harmful intent is expressed through alternative multimodal realizations. We identify reformulation robustness as an overlooked dimension of steering safety and demonstrate that low harmful response rates on canonical benchmarks do not necessarily translate into robust protection against intent-preserving textual, visual, and joint reformulations.

2. Developing an efficient framework to expose this vulnerability. Building on this finding, SteerProbe formulates reformulation selection as a shared calibration problem. Its candidate library, input-conditioned selector, and active calibration jointly enable feedback collected from a limited set of requests to guide one-shot reformulation selection on unseen requests. While the individual components build on established techniques, their integration addresses the specific challenge of efficiently evaluating reformulation robustness under a limited target-query budget.

W2+Q3. Theroretical analysis. More analysises are needed to explain the insights into the effectiveness of the proposed method.

Re. Proposition 1 characterizes when reformulation-induced representation changes can exceed the compensation capacity of steering defenses, while our empirical findings translate this theoretical insight into the design of SteerProbe.

1. Theoretical insight and connection to reformulations. Proposition 1 establishes a sufficient condition under which a reachable reformulation induces a safety-margin reduction that exceeds the original safety margin and the maximum admissible steering compensation. Specifically, uρu_\rho represents the representation shift induced by a reformulation, while sD(z,ρ)s_D(z,\rho) captures the corresponding change in steering correction. Different textual, visual, and joint reformulations can induce different representation shifts, while different steering mechanisms provide different compensation capabilities. This explains why alternative expressions of the same intent may affect steering protection differently across inputs and defenses.

2. Implications for SteerProbe design. The theoretical analysis and empirical observations motivate the three components of SteerProbe:

  • Candidate library: The reachability condition motivates exploring diverse textual, visual, and joint reformulations, rather than relying on a single transformation.
  • Input-conditioned selector: Since the induced representation shifts and steering corrections depend on the input and defense, the effectiveness of a reformulation may vary across requests. Our empirical observations motivate selecting candidates according to request-specific features rather than using a fixed global transformation.
  • Active calibration: Proposition 1 characterizes surrogate-level failure conditions but does not directly predict actual harmful responses or identify the most effective reformulation. We therefore use target-model feedback to learn reformulation effectiveness, while active acquisition prioritizes informative queries under a limited budget.

We add this theoretical-to-design connection after Proposition 1 in Section 3 to clarify how the theoretical findings and empirical observations motivate SteerProbe.

AuthorsOpenReview ↗

Response to Reviewer KFgH (Part: 2/3)

W3+Q1. Comparison with existing black-box jailbreak attacks.

Re. Following your suggestion, we extend our evaluation with three representative black-box jailbreak baselines: Best-of-N (BoN) [R1], PAIR [R2], and TAP [R3]. We conduct all comparisons under equivalent total target-query budgets, including calibration queries. Specifically, SteerProbe uses 500 shared calibration queries plus one final query per test request, resulting in 668 total target queries for MM-SafetyBench. The competing methods are allocated the same total number of target queries for per-request online search. Importantly, under this matched budget, the baseline methods need to allocate multiple queries (up to four) to individually optimize each test request, whereas SteerProbe reuses the shared calibration feedback and requires only one target-model query per unseen request. This highlights the advantage of cross-request feedback sharing over independent per-request search.

As shown in Table R1, SteerProbe achieves the highest HR across both evaluated defenses, outperforming all three black-box baselines. Specifically, it achieves 27.62% HR against NullSteer and 19.64% against L2S, demonstrating the effectiveness of its shared calibration and input-conditioned reformulation selection under a limited target-query budget.

We include the details and results in the Appendix. D5 in the revised paper.

Table R1. Comparison with existing black-box jailbreak baselines under matched target-query budgets. HR (%) is reported.

DefenseBoNPAIRTAPSteerProbe
NullSteer3.579.526.5527.62
L2S10.717.747.1419.64
CMRM_data1.790.602.987.14

[R1]: Hughes, John, et al. "Best-of-n jailbreaking." NeurIPS, 2025.

[R2]: Chao, Patrick, et al. "Jailbreaking black box large language models in twenty queries." 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 2025.

[R3]: Mehrotra, Anay, et al. "Tree of attacks: Jailbreaking black-box llms automatically." NeurIPS, 2024.

W4. The generalization to unseen attack templates, different safety distributions, or other multimodal tasks remains unclear.

Re. We clarify that SteerProbe is designed to enhance existing multimodal jailbreak attacks through reformulation selection, rather than to develop a new source attack or a universal attack transferable across arbitrary models and defenses.

  1. Rationale of the current evaluation setting. Our threat model assumes that an attacker already has an existing multimodal attack pipeline and targets a VLM protected by a specific safety steering defense. Our objective is to improve this existing attack by reusing feedback from previously evaluated requests to select effective reformulations for unseen requests. Therefore, we keep the source attack construction, target VLM, and defense configuration fixed during calibration and testing. This design isolates the effectiveness of cross-request reformulation selection, rather than confounding it with changes in attack algorithms or defense mechanisms.

  2. Additional transfer evaluations. Following your suggestion, we conduct three transfer experiments to evaluate whether a selector calibrated on one source domain can be reused in another without additional target-domain calibration: (1) Cross-attack-construction: Transfer between SD, TYPO, and SD+TYPO within MM-SafetyBench. (3)Cross-category: Transfer between two disjoint groups of the 13 safety categories (A→B and B→A). (3) Cross-benchmark: Transfer between MM-SafetyBench (SD+TYPO) and HADES (VisCRA).

Original denotes the HR obtained by directly applying the source attack without SteerProbe; Transfer denotes the HR obtained using the frozen source-domain selector on the target domain, without additional calibration. The results in Table R2 show that transferred selectors consistently improve HR over the original attacks across all evaluated settings, although the improvement varies across transfer directions and defenses. This demonstrates that SteerProbe retains useful reformulation-selection capability beyond its original calibration domain. We include the details and results in the Appendix. D6 in the revised paper.

Table R2. Transferability of SteerProbe across attack constructions, safety-category distributions, and benchmarks. Each entry reports HR (%) as Original → Transfer.

Transfer settingSource → TargetNullSteerL2S
ConstructionSD+TYPO → SD0.60 → 5.360.60 → 5.95
ConstructionSD+TYPO → TYPO1.19 → 12.502.38 → 14.29
ConstructionSD → SD+TYPO4.17 → 16.672.98 → 11.90
ConstructionTYPO → SD+TYPO4.17 → 11.902.98 → 18.45
Safety categoryA → B5.95 → 22.622.38 → 13.10
Safety categoryB → A2.38 → 19.053.57 → 16.67
BenchmarkMM → HADES7.00 → 16.0012.00 → 20.00
BenchmarkHADES → MM4.17 → 11.902.98 → 10.71
AuthorsOpenReview ↗

Response to Reviewer KFgH (Part: 3/3)

Q2. The performance gains or which benefit most to the final results, more discussion and analysis could be added. library, selector or calibration.

Re. Following your suggestion, we conducted additional ablation studies to distinguish the contributions of the candidate library and learned selection. Specifically, we compare the original inputs without reformulation, random candidate selection without learned feedback, and the full SteerProbe framework. We include the details and results in the Appendix. D7 in the revised paper.

Table R3. Ablation study of the candidate library and learned selection on Qwen2-VL-7B / MM-SafetyBench. HR (%) is reported.

Defensew/o libraryw/o learned selectorFull
L2S8.9317.2619.05
NullSteer8.3322.0232.74
CMRM2.985.3618.45

The results reveal defense-dependent contributions. For L2S, the library provides a substantial gain, while learned selection adds 1.79 percentage points. For NullSteer, the respective gains are 13.69 and 10.72 points. In contrast, for CMRM, random reformulation provides only a 2.38-point gain, whereas learned selection contributes an additional 13.09 points. These results indicate that the relative importance of the candidate library and learned selection varies across steering defenses, with no single component consistently dominating the overall performance.


Thank you again for your time and feedback. We sincerely hope these clarifications help address your concerns.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer KFgH,

Thank you for the detailed questions. To make the discussion easier, we would like to briefly summarize the main additions in our rebuttal.

  1. We have clarified the central contribution of SteerProbe in identifying reformulation robustness as an overlooked vulnerability of existing steering defenses, and strengthened the connection between our theoretical analysis and method design (W1/W2).
  2. We have also added comparisons against BoN, PAIR, and TAP (W3/Q1), cross-attack, cross-category, and cross-benchmark transfer experiments (W4), as well as studies to analyze the contributions of the candidate library and learned selector (Q2).

As the discussion deadline approaches, we would be happy to clarify any remaining questions. If your concerns have been resolved, we would appreciate your reconsideration of the rating. Thanks again!

Best Regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion

Dear AC,

We sincerely thank you for your time and effort throughout the review process, as well as all reviewers for their constructive feedback. For ease of reference, we summarize the main contributions, reviewers' feedback, our responses, and the outcome of the discussion.


Our contributions and recognized strengths

Our main contribution is to reveal an overlooked vulnerability of existing safety steering defenses (i.e., a reformulation robustness gap) and introduce SteerProbe to investigate this vulnerability. We are grateful that reviewers recognized the following strengths:

  • Practical significance and interesting perspective in revealing a weakness in existing safety steering defenses (Reviewers KFgH, risq, mRme).
  • Practical attack setting that requires no access to model parameters, gradients, or hidden states (Reviewers KFgH, mRme).
  • Well-motivated and efficient methodology (Reviewers KFgH, mRme).
  • Comprehensive evaluation across three VLM backbones, three steering defenses, and two multimodal benchmarks (Reviewers KFgH, risq, mRme).

Addressing concerns

During the rebuttal, we addressed the main concerns through the following clarifications and additional experiments:

  • Novelty and theoretical analysis (KFgH W1/W2/Q3, risq W3). We clarified the paper's contributions, theoretical assumptions, and the connection between Proposition 1 and our empirical findings.
  • Evaluation reliability and baseline comparisons (risq W1, mRme W1/Q1, KFgH W3/Q1). We verifies our method using an independent multimodal judge and also added comparisons with Best-of-N, PAIR, and TAP under same query budgets.
  • Generalization and experimental settings (KFgH W4, risq W2). We added cross-attack, cross-category, and cross-benchmark transfer experiments, and clarified our hyperparameter selection.
  • Ablations and design choices (KFgH Q2, mRme W2/W3/Q2/Q3). We conducted additional ablations to examine the contributions of the candidate library, learned selector, and active calibration. We also analyzed reformulation types and candidate library size.
  • Efficiency analysis (mRme W4). We measured calibration overhead and target-model query costs, showing that local selector training is lightweight.

Discussion outcome

Both reviewers who provided follow-up feedback raised their ratings:

  • Reviewer KFgH: No follow-up response
  • Reviewer risq: 2 —>3
  • Reviewer mRme: 3 —>4

We believe the additional analyses and experiments have substantially strengthened our work and addressed the main concerns raised during the review.

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.