- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
- REBASE addresses the gap between recognizing harmful multimodal inputs and producing appropriate refusals. It extracts harmful/benign activation prototypes and a refusal/compliance steering vector, then uses a standardized projection and asymmetric tanh coefficient to adjust generation without updating model weights. Experiments on LLaVA-1.5-7B and Qwen2-VL-7B report improved attack success rate (ASR), benign non-refusal and MM-Vet performance, with cross-dataset transfer and steering-magnitude ablations. The central claims are that directional misalignment explains safety failures and that correcting it requires negligible inference overhead.
Critical strengths
- Table 1 reports a meaningful improvement in the joint safety/utility trade-off. On LLaVA/VLGuard, ASR decreases from 22.85% to 2.04%, while benign non-refusal remains 99.28%; on Falcon, REBASE improves this latter metric from AutoSteer's 75.96% to 89.93% at similar ASR. Automating Steering for Safe Multimodal Large Language Models trains a safety prober and refusal head, whereas REBASE uses activation statistics. Obtaining comparable protection with this simpler controller would be valuable if the matched evaluation is confirmed.
- I find the transfer and magnitude experiments useful. Table 3 transfers vectors in both directions between VLGuard and Falcon, with benign-score losses no larger than 2.63 points, while §4.6.1 tests continuous magnitude against a sign-based controller. Table 2 additionally checks answer correctness on MM-Vet instead of relying entirely on refusal rates. Together these experiments provides evidence that the method can preserve useful behavior, even though they do not yet isolate every proposed mechanism.
Critical weaknesses
- Eq. 3 changes input harmfulness and response behavior together, so the extracted direction and PCA plots do not establish causal misalignment. Programming Refusal with Conditional Activation Steering already conditions refusal and SafeSteer: Adaptive Subspace Steering for Efficient Jailbreak Defense in Vision-Language Models adapts its strength; their text-only and classifier-based designs differ, but missing matched comparisons leave REBASE’s specific contribution unclear.
- Appendix B includes safe-image/unsafe-instruction cases, but the released VLGuard loader excludes them; Falcon evaluation also filters by reference-response content (
prompting_with_steering.py:113;generate_vectors.py:198–208,293–347). These selection rules leave Table 1’s coverage and baseline comparability uncertain until the actual evaluated IDs and subgroup counts are provided. - I could not reconcile §4.5’s single-pass claim with
prompting_with_steering.py:220–239,431–453: it probes, resets, then generates without passing a cache. The timer excludes the probe, so the released implementation does not substantiate Figure 8’s 0.01–0.02-second overhead; end-to-end measurements from the actual experiment are needed. - Appendix A counts gibberish as refusal and substantive non-refusal as helpful, while the safety judge never sees the image. MM-Vet only partly addresses this; unlike the separate outcomes in Refusal in Language Models Is Mediated by a Single Direction, the reported metrics cannot distinguish coherent refusal, degeneration, and harmful compliance.
- Table 3 tests dataset transfer, but defense-aware attacks, as evaluated in Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks, are absent. Falcon ASR gains over AutoSteer are only 0.08/0.04 percentage points without paired intervals, leaving adaptive robustness and ranking uncertain despite larger benign-utility gains.
Questions for authors
- Required clarification: Which exact sample IDs and retained subgroup counts produced Table 1, and were the VLGuard and Falcon filters above applied identically to every baseline?
- Required clarification and validation: Which code produced Figure 8, where is the probe cache reused, and what overhead remains when timing the complete defended request with output lengths controlled?
- Required evidence: Do matched-input response contrasts and full-dimensional interventions support the misalignment explanation, and does REBASE retain an advantage over matched-budget CAST and SafeSteer?
- Required validation: How often are reported “refusals” incoherent or non-actionable harmful responses, and how do image-aware human judgments change ASR, benign correctness and method rankings?
- Required validation: What are the paired confidence intervals and defense-aware attack results, including stablity across extraction samples? These would materially affect confidence in the claimed robustness rather than simply extend the application scope.
2 comments
Warm Reminder - One Day Left in the Discussion Period
Dear Reviewer WqRc,
As the discussion phase ends in less than one day, we kindly ask that you let us know whether we have addressed your concerns, and if so, we would appreciate it if you would consider raising the score. Thank you for your time and effort.
Best regards,
Authors
Dear Reviewer WqRc,
We would greatly appreciate it if you could let us know whether our responses have resolved your concerns or if there is anything else we can clarify. Thank you for your time and effort.
Best regards,
Authors