- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the robustness of steering defenses in VLMs and identities reformulations can substantially increase harmful response rates. Based on this, this paper proposes SteerProbe, an output-only black-box attack that learns to select effective reformulations for unseen requests. Experiments across three VLM backbones, two multimodal jailbreak benchmarks, and three steering defenses show that SteerProbe increases the average harmful. Overall, this paper identity a weakness of current steer-based defenses and provides an interesting direction for future research.
Critical strengths
- The research problem holds practical significance. This paper focuses on the safety robustness of "Activation Steering" when faced with semantically preserved variations in multimodal inputs. This perspective facilitates a more comprehensive understanding of the actual safety boundaries of "Steering Defense."
- The attack scenario is highly practical. SteerProbe employs an "output-only" black-box threat model, requiring no access to the target model’s internal parameters, gradients, or hidden states.
- The methodology is clearly structured and accounts for a limited query budget. The authors decompose the process into distinct modules utilizing lightweight logistic regression to facilitate feedback transfer.
- The experimental evaluation is thorough. The study assesses performance across three different backbones, three distinct defenses, and two multimodal attack benchmarks. The paper also analyzes the impact of several key factors, and possible defenses.
Critical weaknesses
-
The safety evaluation metrics are not enough. This paper mainly uses Llama Guard 3-8B to determine the harmfulness of the model output, and uses Harmful Rate (HR) as the core indicator of attack effectiveness. However, some answers that contain sensitive content but do not actually satisfy malicious requests may affect the final evaluation results. It is recommended that the author add attack success rate evaluation based on a general-purpose LLM Judge (such as GPT-4o), taking into account both the original request and the generated reply, to further verify the reliability of the current experimental conclusions.
-
The design motivation of Logistic Regression as a selector is not sufficient. This paper uses a lightweight Regularized Logistic Regression Model to predict the attack effects of different Reformulations. However, this model is essentially a linear classifier and may struggle to fully capture the complex nonlinear relationships between input features and different rewriting operations. The authors should further explain the reasons for choosing linear models over other predictive models. Furthermore, could using a simple MLP or other lightweight nonlinear model improve the selection of candidate rewrites? It is recommended to supplement relevant comparative experiments to verify the rationality of the current model design.
-
The analysis of the impact of different Reformulation types is still insufficient. This article mainly considers predefined text rewriting, image transformation and the combination of the two, but how different rewriting categories affect the robustness of Steering Defense is still lacking a more detailed analysis. For example, are there significant performance differences between Direct Grounding, Phrase Injection, Reference Resolution, and Instruction Framing in text rewriting? Are these differences consistent across different Steering Defenses? Furthermore, current candidate rewrites are mainly based on fixed templates and conventional image processing operations, and their coverage is relatively limited. The authors could further discuss the basis for the selection of existing rewriting operations, and whether it is possible to further improve the effectiveness of the attack and enhance the generality of the method when extending the method to more semantically preserving text or visual changes.
-
There is a lack of sufficient analysis of the training and calibration costs of the method. This paper highlights that SteerProbe requires only one target model query during the testing phase, but the training and calibration overhead of the method is still not fully discussed. Specifically, SteerProbe needs to collect target model feedback through multiple rounds of Active Calibration and repeatedly update the Logistic Regression Selector while performing uncertainty estimation and candidate selection. However, the paper has not yet reported the actual computational cost and running time of these steps. The authors are recommended to supplement the analysis of Training/Calibration Time and overall computational overhead to further evaluate the actual efficiency and scalability of this method.
Questions for authors
- In addition to the Harmful Rate (HR) based on Llama Guard 3-8B, have the authors considered using other LLM-based judges (such as GPT-4o) for evaluation assessing the actual attack success rate by analyzing both the original malicious requests and the model responses?
- What were the primary considerations behind choosing Regularized Logistic Regression over non-linear models like a simple MLP? Could more complex models yield further improvements in attack effectiveness given the same calibration budget?
- Are there significant differences in how various types of Text Reformulation (e.g., Direct Grounding, Phrase Injection, and Reference Resolution) affect different Steering Defenses? Furthermore, given that the current method relies heavily on predefined text templates and image processing operations, have the authors considered extending the approach to include more semantic-preserving reformulation techniques?
4 comments
Response to Reviewer mRme (Part: 2/2)
W3+Q3. The analysis of the impact of different Reformulation types is still insufficient.
Re: Following your suggestion, we add a finer-grained analysis of reformulation families and discuss the scope of the current candidate library. We include the details and results in the Appendix. D4 in the revised paper.
1. Effects of different reformulation types. We group the textual candidates into four families: Direct Grounding, Phrase Injection, Reference Resolution, and Instruction Framing. Table R3 reports, for each defense, the proportion of successful final attacks whose selected textual candidate belongs to each family. Thus, the entries describe the composition of successful attacks rather than the standalone HR of each family.
Table R3. Distribution of successful final attacks across textual reformulation families (%).
| Reformulation family | L2S | NullSteer | CMRM_data |
|---|---|---|---|
| Direct Grounding | 31.25 | 21.82 | 25.81 |
| Phrase Injection | 37.50 | 45.45 | 22.58 |
| Reference Resolution | 18.75 | 18.18 | 29.03 |
| Instruction Framing | 12.50 | 14.55 | 22.58 |
2. Effect of candidate-library size. We further vary the number of textual candidates while keeping the image-transformation set, calibration budget, selection procedure, and evaluation requests fixed. Table R4 shows that the relationship between library size and HR is not monotonic. These findings indicate that a larger candidate pool can increase coverage, but under a fixed calibration budget it can also make effective candidates harder to identify reliably.
We also agree that broader semantic coverage is an important direction for future work. Future extensions could explore more diverse and semantically preserving reformulations to further expand the coverage of potential vulnerabilities. Our current work primarily aims to identify and systematically expose the limitations of existing safety steering defenses under multimodal reformulations, rather than exhaustively exploring the entire space of possible transformations.
Table R4. HR (%) under different textual candidate-library sizes.
| Defense | K = 6 | K = 12 | K = 18 | K = 24 |
|---|---|---|---|---|
| L2S | 17.26 | 11.31 | 19.05 | 17.26 |
| NullSteer | 33.93 | 32.74 | 32.74 | 27.38 |
| CMRM_data | 6.55 | 8.93 | 18.45 | 10.12 |
W4. There is a lack of sufficient analysis of the training and calibration costs of the method.
Re: Following your suggestion, we provide additional measurements to quantify the computational overhead of active calibration and selector fitting. Specifically, the active candidate acquisition process takes 73.74 seconds across 27 rounds, averaging 2.73 seconds per round. After collecting the 500 calibration observations, fitting the final regularized logistic regression selector takes only 0.114 seconds. These results demonstrate that the local acquisition and selector-fitting computations are lightweight. The main calibration cost comes from collecting target-model feedback, while the selector itself can be fitted efficiently. We will include these measurements in the revised manuscript to provide a clearer account of SteerProbe's computational overhead. We include the details and results in the Appendix. D3 in the revised paper.
Table R5. Computational overhead of active calibration and selector fitting.
| Computational component | Measured time |
|---|---|
| Active candidate acquisition (27 rounds) | 73.74 s |
| Average acquisition time per round | 2.73 s |
| Final logistic regression fitting | 0.114 s |
Thank you again for your time and feedback. We sincerely hope these clarifications help address your concerns.
Thank you for your detailed response and additional experimental results. At present, my major concerns are solved through these additional experiments and clarifications. The added analysis strengthens the paper's reliability and quality. I will incerese my rating from 3 to 4.
Dear Reviewer mRme,
Thank you very much for your positive feedback and for raising the score to clear acceptance! It encourages us a lot! We sincerely appreciate your recognition of our work's contributions, and your constructive feedback helps us improve the quality of our work. Thanks!
Best regards,
The Authors
Response to Reviewer mRme (Part: 1/2)
Thanks for your careful review. We sincerely hope the following clarifications can address your concerns. All changes in the revised PDF are highlighted in red.
Re: Following your suggestion, we further employ DeepSeek-V4-Flash as an independent multimodal judge. Specifically, the judge receives the original image–text input together with the complete response generated by the target VLM and determines whether the response provides substantive harmful information that follows and fulfills the original malicious intent. We use a binary Yes/No judgment and count only affirmative responses as successful attacks. We report the resulting ASR in the following Table. The results are consistent with Table 1 in the paper.
We include the details and results in the Appendix. D1 in the revised paper.
Table R1. ASR (%) evaluated by DeepSeek-V4-Flash.
Re: We choose regularized logistic regression for its data efficiency, low computational cost, and tractable uncertainty estimation under limited calibration feedback. Our 508-dimensional feature representation captures request-specific information and reformulation characteristics, enabling input-conditioned candidate selection. Moreover, logistic regression supports efficient iterative fitting and Laplace-based uncertainty estimation, making it suitable for active calibration.
Following your suggestion, we further compare logistic regression with a one-hidden-layer MLP (32 ReLU units), using identical features, 500 calibration observations, and test requests. Logistic regression achieves a 1.79-percentage-point higher HR while requiring approximately 627× less fitting time. These results support its effectiveness and computational efficiency under our limited calibration budget. We include the details and results in the Appendix. D2 in the revised paper.
Table R2. Comparison between the regularized logistic regression selector and a lightweight nonlinear MLP on Qwen2-VL-7B / MM-SafetyBench.