acceptodds
Under review as a conference paper at ICLR 2027

GenShield: Generalizable Visual Prompts for VLM Jailbreak Defense via Dual-View Adversarial Training

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable multimodal capabilities, yet they remain highly vulnerable to jailbreak attacks that bypass their safety alignment and elicit harmful outputs. Defensive visual prompts have recently emerged as a lightweight and plug-and-play defense interface for VLMs. However, existing defensive visual prompts are typically trained against a fixed, hand-curated set of attacks, causing them to overfit to specific attack patterns and scene statistics seen during training, and consequently to transfer poorly to unseen attack strategies or novel visual scenes. We present GenShield, a micro-region adversarial training framework that perturbs only about 2% of the visual input to produce generalizable defensive visual prompts against VLM jailbreak attacks, guided by a dual-view analysis of attack-space coverage (attack view) and prompt information dependency (defense view). From the attack view, we employ beam search over base attack strategies to synthesize the strongest composite attacks in the inner loop, densely covering the attack manifold. From the defense view, we introduce mutual information regularization in the outer loop to strengthen the dependence between the visual prompt and prompt-induced decision changes while reducing their image dependence, promoting consistent defensive effects across visual scenes. Theoretically, we further establish a unified generalization bound for the resulting defensive visual prompts under the dual-view design. Extensive experiments across multiple VLM architectures and attack methods demonstrate that GenShield achieves strong cross-attack and cross-scenario generalization, while largely preserving benign utility and maintaining high visual fidelity.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
vision-language models, jailbreak defense, adversarial training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eYtnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes GenShield, a generalizable visual prompt defense against vision-language model (VLM) jailbreak attacks. To improve cross-attack generalization, it employs beam search to construct strong composite attacks during adversarial training. To enhance cross-scene generalization, it introduces dual-direction mutual information regularization that strengthens the dependence between visual prompts and prompt-induced decision changes while reducing their dependence on image content. The framework optimizes a universal visual prompt within approximately 2% of the image without modifying VLM parameters. Experiments on multiple VLMs and benchmarks demonstrate promising defense effectiveness and generalization while largely preserving benign utility and visual fidelity.

Critical strengths
  1. This paper addresses generalization from both attack-space coverage and scene dependency perspectives, combining compositional attack search with mutual information regularization in a well-motivated framework.
  2. The dual-direction mutual information regularization based on prompt-induced decision changes is an interesting design, achieving promising cross-attack generalization while modifying only about 2% of the image and largely preserving benign utility.
Critical weaknesses
  1. The overall methodological novelty is limited, as the framework largely builds upon SafeLogo [1], while adversarial prompt optimization [2] and search-based attack generation [3] have been extensively studied.
  2. The necessity of the proposed mutual information regularization remains unclear, as comparisons with simpler alternatives, such as decision-change variance minimization or cross-scene consistency regularization, are missing.
  3. This paper claims that compositional beam search improves attack-space coverage but provides no direct evidence of increased attack diversity or coverage, nor comparisons with simpler composition strategies under matched query budgets.
  4. The provided training script disables online compositional attack search, and the required external search implementation is not included in the released code. Additionally, the code introduces training components not clearly described in the paper, raising concerns about implementation consistency and reproducibility.

[1] Duan, Z., et al. SafeLogo: Turning Your Logos into Jailbreak Shields via Micro-Regional Adversarial Training.

[2] Mo, Y., et al. Fight Back Against Jailbreaking via Prompt Adversarial Tuning.

[3] Mehrotra, A., et al. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically.

Questions for authors
  1. Under matched attack-query and training budgets, how does compositional beam search compare with SafeLogo's adaptive attack selection [1], random strategy composition, and search-based alternatives inspired by TAP [2]?
  2. Can the authors provide quantitative evidence that compositional search improves attack-space coverage rather than simply generating stronger or more numerous attacks?
  3. Why are InfoNCE and conditional vCLUB preferable to simpler decision-change consistency or variance regularization, and can the authors provide corresponding comparisons?
  4. Can the authors clarify whether the experiments use the additional safety text prefix and benign refusal penalty found in the released code, and provide the complete compositional attack search implementation?

[1] Duan, Z., et al. SafeLogo: Turning Your Logos into Jailbreak Shields via Micro-Regional Adversarial Training.

[2] Mehrotra, A., et al. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically.

10 comments

Thank you for your helpful comments and suggestions. We have added the attack-construction strategy comparison to Section 6.4, the textual strategy coverage analysis to Appendix E.1, and comparisons with simpler regularizers to Appendix E.4. We have also clarified the safety text prefix, benign refusal penalty in the implementation details appendix. The corresponding additions are highlighted in blue for ease of reference.

W1-Methodological novelty. The overall methodological novelty is limited, as the framework largely builds upon SafeLogo [1], while adversarial prompt optimization [2] and search-based attack generation [3] have been extensively studied.

Response. Thank you for encouraging us to clarify our methodological contribution. GenShield jointly addresses limited training attack coverage and dependence on visual scenes, two factors that the cited methods do not explicitly target together, to improve generalization across attacks and visual scenes. Our compositional search and MI regularization can also be incorporated into other visual-prompt defenses. We clarify the distinctions from the three cited works below.

  1. Distinction from SafeLogo [1]. We adopt SafeLogo’s adversarial training framework because its localized visual prompt offers a practical balance between defense effectiveness and benign utility. SafeLogo’s key insight is to balance defense effectiveness and benign utility, while GenShield builds on these advantages to explicitly address attack coverage and scene dependence, two factors affecting defense generalization. Our compositional search and MI regularization can also be incorporated into other trainable visual-prompt defenses, not just SafeLogo.

  2. Distinction from Prompt Adversarial Tuning [2]. PAT optimizes a textual defense prefix using adversarial and benign objectives. These objectives do not explicitly model attack-strategy coverage or visual-scene dependence. GenShield introduces training mechanisms explicitly targeting these two generalization factors: compositional search broadens training attack coverage, while MI regularization encourages decision changes to depend more on the visual prompt and less on the visual scene. Together, they target generalization across both attacks and visual scenes.

  3. Distinction from Tree of Attacks [3]. TAP iteratively generates and refines attack prompts using model feedback to find a successful jailbreak. GenShield explicitly searches over combinations of attack strategies, generating composed attacks for visual-prompt training. Table R2 shows that these composed attacks improves attack-space coverage. This provides evidence that explicit strategy composition broadens the attack coverage used to train the defense.

References

[1] Duan, Z., et al. SafeLogo: Turning Your Logos into Jailbreak Shields via Micro-Regional Adversarial Training. CVPR, 2026.

[2] Mo, Y., et al. Fight Back Against Jailbreaking via Prompt Adversarial Tuning. NeurIPS, 2024.

[3] Mehrotra, A., et al. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. NeurIPS, 2024.

W2-Necessity of MI regularization. The necessity of the proposed mutual information regularization remains unclear, as comparisons with simpler alternatives, such as decision-change variance minimization or cross-scene consistency regularization, are missing.
Q3-Comparison with simpler regularizers. Why are InfoNCE and conditional vCLUB preferable to simpler decision-change consistency or variance regularization, and can the authors provide corresponding comparisons?

Response. Thank you for suggesting these comparisons to clarify the role of MI regularization. Both simpler alternatives encourage similar decision changes across samples, but neither explicitly strengthens the dependence of these changes on the visual prompt. Our MI formulation combines two objectives: InfoNCE encourages stronger prompt dependence, while conditional vCLUB reduces visual scene dependence given the prompt. This “push–pull” design aims to reduce reliance on scene-specific cues, supporting generalization to new scenes without requiring identical adjustments across inputs. Together with the safety and benign objectives, this formulation achieves a better balance between defense effectiveness and benign utility than the simpler alternatives in Table R1.

  1. Compared with decision-change variance minimization.

    Decision-change variance minimization penalizes variation in decision changes across samples, encouraging similar adjustments regardless of intent. However, different inputs may need different adjustments to reach safe outcomes. Pulling these adjustments toward a common value may produce a compromise that works on training images but provides insufficient correction in more challenging unseen scenes. Our MI formulation instead encourages decision changes to depend more on the visual prompt and less on the visual scene, without requiring the same amount of adjustment for every input.

  2. Compared with cross-scene consistency regularization.

    Restricting consistency to scenes sharing the same harmful intent makes the constraint more targeted. However, these scenes may still require different amounts of correction. Encouraging similar changes can therefore lead to unnecessarily large adjustments for samples needing only minor correction. Since the same visual prompt is also applied to benign inputs, this tendency may interfere with legitimate responses. Our MI formulation instead regulates prompt and scene dependence without directly matching the adjustment magnitudes.

  3. Experimental comparison.

    To evaluate these design choices empirically, we compare both alternatives with our MI formulation, training without MI, and the undefended model. For cross-scene consistency regularization, we use the three images (SD, SD_TYPO, and TYPO) corresponding to each harmful intent in MM-SafetyBench and minimize the variance of their decision changes. Table R1 shows that our MI formulation better balances defense effectiveness and benign utility.

    Table R1. Comparison with simpler decision-change regularizers. Cross-attack evaluation uses GPTFuzzer (ASR@5), and cross-scene evaluation uses PAIR on VLGuard. ASR is reported in % (↓); higher MM-Vet is better (↑).

    ConfigurationCross-attack ASR@5Cross-scene ASRMM-Vet
    No defense42.0029.4136.9
    Without MI33.0015.1634.7
    Decision-change variance minimization29.0015.6135.3
    Cross-scene consistency regularization29.0013.5734.2
    InfoNCE + conditional vCLUB29.0012.6736.6
Reviewer_qgghOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GenShield, a framework to learn universal visual prompts that defend against jailbreak attacks against VLMs, combining two main components, compositional attack search and mutual information regularisation. The results show that GenShield achieves comparable defense performance compared to existing methods under a smaller perturbation budget and while preserving more benign utility overall.

Critical strengths

The issue of lightweight and easily adaptable jailbreak defense that the paper focuses on is important and interesting in my opinion.

Critical weaknesses
  1. One of the key motivations for prompt-level defense that the paper states is that they are “plug-and-play, and model-agnostic”. Yet, the main experiments do not reflect this setting, training separately on each VLM. Table 14 in the appendix shows that transferring the visual prompts to a commercial black-box VLM results in only minor improvement in ASR compared to no defense, and does not outperform DAVSP.

  2. The paper focuses on VLMs as they are more vulnerable due to the expanded attack surface exposed by the visual modality, but the jailbreak methods tested do not match this threat model and only modify the textual input. The jailbreak methods discussed in the related works section are also relatively outdated. The paper should consider more recent jailbreak methods specifically designed for VLMs, such as JPS [1], UltraBreak [2], and Visual Cipher [3].

[1] Chen, Renmiao, et al. "Jps: Jailbreak multimodal large language models with collaborative visual perturbation and textual steering." Proceedings of the 33rd ACM International Conference on Multimedia. 2025.

[2] Cui, Kaiyuan, et al. "Toward universal and transferable jailbreak attacks on vision-language models." The Fourteenth International Conference on Learning Representations. 2026.

[3] Azulay, Aharon, et al. "Jailbreaking Vision-Language Models Through the Visual Modality." Proceedings of the 43rd International Conference on Machine Learning. 2026.

Questions for authors
  1. Which model is used as the frozen VLM in training to obtain the defensive prompt(s) transferred to the commercial VLM in Table 14? Could you also test this transfer setting on the open-source models?

  2. Could you provide results on more recent jailbreak attacks targeting the visual modality?

  3. Since the method involves a compositional attack search in the optimisation loop, what is the computational cost compared to prior methods?

9 comments

Thank you for your helpful comments and suggestions. We have updated the Related Works section to include JPS, UltraBreak, and Visual Cipher, and added evaluations against these attacks in Section 6.5. Supplementary cross-model transfer results and attack-construction cost measurements are provided in Appendix E.4. The corresponding additions are highlighted in blue in the revised manuscript for ease of reference.

W1(a)-Motivation and scope. One of the key motivations for prompt-level defense that the paper states is that they are “plug-and-play, and model-agnostic”. Yet, the main experiments do not reflect this setting, training separately on each VLM.

Response. We apologize for the unclear wording. Our primary motivation is to improve generalization across attacks and visual scenes, as illustrated in Figure 1 of the main paper. “Plug-and-play” and “model-agnostic” describe practical properties of visual-prompt defenses, which is the research background rather than our central research objective. Specifically, “plug-and-play” refers to directly applying the learned visual prompt at inference, while “model-agnostic” means that visual prompts can be used as a defense across different VLMs.

W1(b)-Transfer to a commercial VLM. Table 14 in the appendix shows that transferring the visual prompts to a commercial black-box VLM results in only minor improvement in ASR compared to no defense, and does not outperform DAVSP.

Response. Thank you for raising this concern about the effectiveness of transfer to a commercial VLM. Our visual prompt is designed to reinforce the VLM’s existing safety behavior. The commercial model already has a relatively low ASR without defense, leaving less room for absolute improvement. GenShield matches DAVSP’s ASR while keeping perturbations bounded at 64/255, whereas DAVSP uses unconstrained border perturbations. Our method can also be configured to use border perturbations without a magnitude constraint. However, this flexibility can come at the cost of visual quality and benign utility. These costs are illustrated by our main-paper experiments: DAVSP yields an SSIM of 0.209 versus 0.983 for GenShield (Table 3), and its MM-Vet score on LLaVA-1.5-13B drops by 0.6 points from the undefended baseline, compared with 0.3 points for GenShield (Table 4). We therefore favor bounded, localized perturbations as a more practical balance between defense effectiveness, visual fidelity, and benign utility.

W2(a)-Threat-model alignment. The paper focuses on VLMs as they are more vulnerable due to the expanded attack surface exposed by the visual modality, but the jailbreak methods tested do not match this threat model and only modify the textual input.
W2(b)-Recent VLM jailbreak attacks. The jailbreak methods discussed in the related works section are also relatively outdated. The paper should consider more recent jailbreak methods specifically designed for VLMs, such as JPS [1], UltraBreak [2], and Visual Cipher [3].
Q2-Evaluation against visual-modality attacks. Could you provide results on more recent jailbreak attacks targeting the visual modality?

Response. Thank you for highlighting this gap in our evaluation. Our initial choice of evaluation setting was based on the shared goal of attacks targeting either visual or textual inputs: triggering unsafe responses through the interaction between visual content and textual instructions. We therefore focused on images containing harmful content paired with optimized text designed to elicit harmful responses based on that content. This setting evaluates a multimodal threat, even though our search optimizes only the textual component. To broaden the evaluation beyond this initial setting, we have now added JPS, UltraBreak, and Visual Cipher, which directly target the visual modality.

  • Evaluation settings.

    Each of the three newly evaluated visual-modality attacks is tested on 300 samples. For comparison, we also include the text-optimized attacks AutoDAN, TAP, and GPTFuzzer from the main-paper evaluation.

  • Results.

    Table R1 shows that GenShield achieves competitive defense performance against these recent visual-modality attacks, although we acknowledge that its gains over existing defenses remain modest.

    Table R1. Defense performance against visual-modality and text-optimized attacks. ASR is reported in % (↓); GPTFuzzer uses ASR@5.

    DefenseJPSUltraBreakVisual CipherAutoDANTAPGPTFuzzer
    No defense84.0029.0054.0012.213.842
    DAVSP76.004.0035.009.66.836
    SafeLogo54.336.6735.339.211.630
    GenShield52.675.3333.333.69.429

    We have updated the Related Works section to include these recent VLM jailbreak methods.

Q1-Source model and cross-model transfer. Which model is used as the frozen VLM in training to obtain the defensive prompt(s) transferred to the commercial VLM in Table 14? Could you also test this transfer setting on the open-source models?

Response. Thank you for requesting this clarification and suggesting the additional transfer evaluation.

  1. The prompt transferred to the commercial VLM in Table 14 was trained on the frozen LLaVA-1.5-13B.

  2. We additionally evaluate direct transfer of the visual prompt trained on LLaVA-1.5-13B to two open-source VLMs, Qwen3.8-27B and Gemma-4-E4B-it. Table R2 shows that GenShield achieves a favorable balance between defense effectiveness and benign utility on both target models.

    • Cross-attack defense. AutoDAN on MM-SafetyBench.
    • Cross-scene defense. PAIR on JailbreakV-28k and PAP on VLGuard.
    • Benign utility. MM-Vet.

    Although DAVSP achieves a lower average ASR on Gemma-4-E4B-it, it incurs a larger loss in benign utility relative to the undefended model than GenShield.

    Table R2. Cross-model transfer to open-source VLMs. ASR is reported in % (↓); higher MM-Vet scores are better (↑). Bold indicates the best average ASR and MM-Vet score among the evaluated defenses for each model.

    Target modelDefensePAIRPAPAutoDANAverage ASR ↓MM-Vet ↑
    Qwen3.8-27BNo defense2.145.436.604.7264.1
    DAVSP1.790.450.000.7557.8
    SafeLogo2.140.000.200.7860.1
    GenShield1.790.000.200.6660.9
    Gemma-4-E4B-itNo defense5.7110.866.607.7257.7
    DAVSP4.6410.631.005.4252.3
    SafeLogo5.3612.901.206.4954.9
    GenShield5.0011.541.606.0556.0

Q3-Computational cost. Since the method involves a compositional attack search in the optimisation loop, what is the computational cost compared to prior methods?

Response. Thank you for raising this question about computational cost. Notably, computational cost is typically not the primary challenge in visual defense prompt optimization, as it is relatively lightweight compared with safety alignment through model fine-tuning. This efficiency is one of the practical advantages of visual defense prompts.

  • The inner-loop search adds cost during training, but the learned prompt can then be reused across attacks without repeating search or retraining for each new attack setting. This allows the additional training cost to be amortized across attack settings. Specifically, compared with SafeLogo-style adaptive selection, GenShield incurs additional attack-construction cost but achieves better cross-attack defense (Table R3).

  • We compare GenShield with SafeLogo-style adaptive selection [1] and a TAP-inspired search implementation [2]. Table R3 shows that our compositional search offers a favorable balance between the cost of constructing attacks during training and the resulting cross-attack defense performance.

    Table R3. Attack-construction cost and cross-attack defense performance. ASRs are from the previously reported defense evaluation. ASR is reported in % (↓).

    MethodAverage construction time (s/sample) ↓Average GPU utilization (%)AutoDANTAP
    SafeLogo-style adaptive selection [1]26.6093.219.211.6
    TAP-inspired search [2]157.0681.156.29.8
    GenShield compositional beam search56.0945.763.69.4
  • Comparison setup.

    • We use SafeLogo-style adaptive selection [1] as a reference for assessing the additional cost and defense benefit of constructing new attacks beyond existing candidates. TAP-inspired search [2] provides an alternative tree-based search approach.
    • All three implementations use LLaVA-1.5-13B as the target model.
    • Average time measures the complete attack-construction or candidate-selection process per input sample.

References

[1] Duan, Z., et al. SafeLogo: Turning Your Logos into Jailbreak Shields via Micro-Regional Adversarial Training. CVPR, 2026.

[2] Mehrotra, A., et al. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. NeurIPS, 2024.

Reviewer_xz9COpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a defense against jailbreak attacks targeting vision-language models (VLMs). Its framework alternates between compositional attack search and defensive visual prompt optimization. During prompt optimization, InfoNCE regularization is introduced to strengthen the dependence between the visual prompt and the decision changes it induces, while conditional vCLUB regularization is used to reduce the dependence of these changes on specific visual scenes.

Critical strengths

1.The incorporation of InfoNCE and vCLUB enhances the novelty of the proposed approach and contributes to the generalizability of visual prompts. 2.The defense introduces only minor modifications to the input images. 3.The proposed method is supported by extensive theoretical analysis.

Critical weaknesses

1.The experiments cover only a small number of relatively dated models. Including more recent VLMs, such as Qwen3.5/3.8 and Gemma-3/4, would strengthen the evaluation. 2.The defense relies on pretrained estimators, but the paper does not examine how their training affects the final defense performance.

Questions for authors

1.Could the authors include a more recent VLM in the experiments? 2.Could the authors evaluate how removing both InfoNCE and vCLUB affects the transferability of the learned visual prompts? 3.Could the authors discuss how estimator training affects the final defense performance?

5 comments

Thank you for your helpful comments and suggestions. We have updated the revised manuscript to reflect the additional evaluations. The opening of the Experiments section now mentions evaluations on four additional VLMs, with detailed settings and results in Appendix E.4. The estimator-pretraining comparison is also included in Appendix E.4 and referenced in Ablation and sensitivity studies. We have added the joint removal of InfoNCE and conditional vCLUB to the existing component ablation table. The corresponding additions are highlighted in blue for ease of reference.

W1-Limited evaluation on recent VLMs. The experiments cover only a small number of relatively dated models. Including more recent VLMs, such as Qwen3.5/3.8 and Gemma-3/4, would strengthen the evaluation.
Q1-Evaluation on recent VLMs. Could the authors include a more recent VLM in the experiments?

Response. Thank you for suggesting this broader evaluation. We have extended the evaluation to four recent VLMs by directly transferring the visual prompt trained on LLaVA-1.5-13B. Table R1 shows that GenShield achieves a favorable balance between defense effectiveness and benign utility across these newer VLMs. Notably, as shown by the MM-Vet decrease relative to the undefended model (Δ in Table R1), DAVSP incurs a larger loss in benign utility than GenShield on Gemma-4-E4B-it, despite achieving a lower average ASR. These results highlight the practical value of our localized visual prompts with bounded perturbations in improving defense while limiting the loss of benign utility.

  • Evaluation settings.

    • Cross-scene defense is evaluated using PAIR on JailbreakV-28k and PAP on VLGuard.
    • Cross-attack defense is evaluated using AutoDAN on MM-SafetyBench.
    • Benign utility is evaluated using MM-Vet.

    Table R1. Defense effectiveness and benign utility on recent VLMs. ASR is reported in % (↓); higher MM-Vet scores are better (↑).

    Model (release)DefensePAIR ↓PAP ↓AutoDAN ↓Average ASR ↓MM-Vet ↑
    Qwen3.5-4B (2026-03)No defense4.297.018.006.4363.6
    DAVSP1.430.230.200.6248.1 (Δ = −15.5)
    SafeLogo0.710.230.200.3848.8
    GenShield1.070.000.000.3651.9
    Qwen3.8-27B (2026-08)No defense2.145.436.604.7264.1
    DAVSP1.790.450.000.7557.8 (Δ = −6.3)
    SafeLogo2.140.000.200.7860.1
    GenShield1.790.000.200.6660.9
    Gemma-3-4B-it (2025-03)No defense36.4332.1326.8031.7956.1
    DAVSP28.5722.8511.4020.9450.8 (Δ = −5.3)
    SafeLogo27.1423.0813.2021.1454.8
    GenShield23.2122.6210.8018.8855.9
    Gemma-4-E4B-it (2026-03)No defense5.7110.866.607.7257.7
    DAVSP4.6410.631.005.4252.3 (Δ = −5.4)
    SafeLogo5.3612.901.206.4954.9
    GenShield5.0011.541.606.0556.0

W2-Estimator training analysis. The defense relies on pretrained estimators, but the paper does not examine how their training affects the final defense performance.
Q3-Impact of estimator training. Could the authors discuss how estimator training affects the final defense performance?

Response. Thank you for suggesting this analysis. Following your suggestion, we added two ablations that limit pretraining to five epochs for either the InfoNCE estimator or the residual conditional vCLUB estimator, and compared them with full pretraining. We find that limiting either estimator’s pretraining to five epochs weakens both cross-attack and cross-scene defense (Table R2). More complete pretraining may yield reliable dependence estimates, providing better guidance for visual-prompt optimization and helping the learned prompt generalize across attacks and visual scenes.

  • Evaluation settings.

    We limit pretraining to five epochs for either the InfoNCE estimator or the residual conditional vCLUB estimator and compare each variant with full pretraining.

    • Cross-attack defense is evaluated using AutoDAN on 100 MM-SafetyBench samples.
    • Cross-scene defense is evaluated using PAIR on 100 JailbreakV-28k samples.

    Table R2. Effect of estimator pretraining on final defense performance. Values are ASR (%) ↓.

    Estimator pretrainingAutoDAN, cross-attackPAIR, cross-scene
    Full pretraining69
    InfoNCE estimator limited to 5 epochs1211
    Residual conditional vCLUB estimator limited to 5 epochs1412

Q2-Joint ablation of InfoNCE and vCLUB. Could the authors evaluate how removing both InfoNCE and vCLUB affects the transferability of the learned visual prompts?

Response. Thank you for suggesting this additional ablation. Building on the existing ablations that remove each term individually, we now evaluate a variant that removes both InfoNCE and conditional vCLUB. We find that removing both InfoNCE and conditional vCLUB weakens transferability across attacks and visual scenes (Table R3). Jointly removing both terms also reduces benign utility. Full GenShield achieves the strongest cross-scene defense while retaining MM-Vet performance close to the best ablation result.

  • Table R3. Individual and joint ablations of InfoNCE and conditional vCLUB. ASR is reported in % (↓); higher MM-Vet scores are better (↑).

    ConfigurationCross-attack ASR@5 ↓Cross-scene ASR ↓MM-Vet ↑
    Without both InfoNCE and conditional vCLUB33.0015.1634.7
    Without InfoNCE29.0014.2536.7
    Without conditional vCLUB30.0013.5736.1
    Full GenShield29.0012.6736.6

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.