acceptodds
Under review as a conference paper at ICLR 2027

FSPO: Fast Adversarial Training against Jailbreak Attacks via Soft Prompt Optimization

Abstract

As large language models (LLMs) have achieved remarkable performance and are increasingly deployed across diverse domains, ensuring their reliability and safety has become critically important. Among various threats, jailbreak attacks, which manipulate model to elicit unsafe or prohibited outputs, have become a major challenge. Although various defensive frameworks have been proposed, most existing defenses are computationally expensive. Moreover, since current defenses do not explicitly account for adaptive attacks, they are easily circumvented by strong adaptive attacks. To address these issues, we propose *Fast Soft Prompt Optimization*, which effectively enhances robustness by relaxing discrete prompt into a continuous probability simplex, called a soft prompt, and adaptively optimizing it. The resulting soft prompt can be directly deployed as a continuous prefix within the model prior to release. We evaluate our methods under two information-rich settings: model-aware but defense-unaware static attacks, and model-and-defense-aware adaptive attacks. We further analyze attention shift mechanisms to provide insights into the underlying robustness improvements.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Jailbreak, AI Safety, Trustworthy AI, Jailbreak Defense, Prompt Tuning, Adversarial Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yhKhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Fast Soft Prompt Optimization (FSPO), a prompt-level jailbreak defense that optimizes a continuous defensive prefix in the vocabulary simplex against an adaptive adversarial suffix. The method is motivated by the cost of existing defenses and by their tendency to be circumvented by adaptive attacks. The authors evaluate FSPO under static and adaptive attack settings and report improved robustness with lower optimization cost than GCG-style baselines.

Critical strengths
  • Clean continuous-relaxation formulation that avoids expensive token-level search.
  • Min-min optimization explicitly accounts for an adaptive attacker.
  • Reported per-step speedup and a deployable pre-installed prefix.
Critical weaknesses
  1. The comparison baseline is outdated and omits the most relevant recent defenses. The main prompt-level baselines are RPO and PAT while more recent and closely related work is neither compared nor discussed: DDPO [1] (AAAI 2026) learns an input-dependent deep prompt in a frozen LLM, and MCD [2] (EMNLP Findings 2025) optimizes a continuous soft safety prompt. Since FSPO's central claim is continuous soft-prompt optimization, the absence of these comparisons makes it difficult to assess its novelty and relative effectiveness. At minimum, the paper should discuss these methods and, where feasible, include them as baselines.

  2. The method depends on the base model's pre-existing safety alignment. FSPO performs well on aligned models such as Llama-2-7B-Chat and gpt-oss-20b, but remains much weaker on the unaligned Vicuna-7B-v1.3: static PAIR ASR is 40% (vs. 0% on Llama) and adaptive PAIR ASR is 78% (vs. 0% on Llama). The paper attributes this to FSPO "recover[ing] and reinforce[ing] the safety behaviors of the base model," which suggests that the method is not learning a standalone defense but is largely reactivating existing refusal capacity. This limits its applicability to weakly aligned or unaligned models and should be analyzed more directly, for example by testing whether the gains disappear when the base model's safety behavior is removed.

[1] Obidov et al. Dynamic Deep Prompt Optimization for Defending Against Jailbreak Attacks on LLMs. AAAI 2026

[2] Li et al. Multilingual Collaborative Defense for Large Language Models. Findings of EMNLP 2025.

Questions for authors

See weaknesses

3 comments

W1. Continuous-defense baselines and constraint motivation

FSPO has lower static ASR than DDPO, DRO and our MCD reconstruction.

DDPO (Obidov et al., 2026) learns input-dependent deep prompts; MCD (Li et al., 2025) uses multilingual collaboration. FSPO learns a continuous prefix (Sections 2.2/5.2).

We evaluate 98 Llama-2-Chat GCG inputs:

DefenseUnsafe /98
DDPO, author-code training11
DRO, released tensor12
MCD, local reconstruction15
FSPO2
  • DDPO uses unmodified author training code.

  • DRO (Zheng et al., 2024) uses the unchanged released 116-vector tensor.

  • MCD is a provisional local reconstruction: missing helpers, data, probes and configurations require different translation, probe-fitting and judging components.

Our GCG inputs provide a more challenging test than the baseline papers' evaluations. Our budget is 500 steps, 256 candidates and top-k256. DRO's released traces contain at most 100 logged iterations; MCD evaluates multilingual harmful queries rather than GCG inputs; DDPO uses a different GCG corpus whose search budget is not recoverable from the inspected paper/release. Stronger attack prompts are a plausible contributor to the published-result gap.

Simplex bounds embedding excursions; entropy sharpens mixtures inside the hull.

  1. Direct Z learns the defense embedding matrix ZZ (20 vectors, dimension 4096), without simplex or entropy constraints. All 20 vectors lie far outside the vocabulary hull. Direct Z produces 22/98 unsafe responses, while ARC-C drops from 39.5% without a prefix to 16.5%, with malformed fragments in 14/25 benign responses. We hypothesize that off-hull embeddings contribute to this utility degradation.

  2. Let EE be the frozen vocabulary-embedding matrix and AA the mixing coefficients. With Z=AEZ=AE, nonnegative unit-sum rows keep each vector in the vocabulary hull.

  3. Entropy sharpens toward one-hot rows, where one coefficient selects one token: mean nearest-one-hot distance falls 0.598→0.180, and the mean largest coefficient rises 0.444→0.866. Sharpening uses fixed factor 0.8.

The ablation fixes coefficient coordinates:

SimplexEntropy factorUnsafe /98ARC-C200
✗✗3224%
✓✗1831%
✓✓237%

In this ablation, adding simplex and then entropy improves both safety and utility.

Our contribution is constrained continuous adversarial training for jailbreak defense.

We formulate prompt-tuning-based jailbreak defense as constrained continuous adversarial training, building on continuous prompt learning (Lester et al., 2021; Li & Liang, 2021) and adversarial training (Madry et al., 2017). RPO/PAT optimize discrete defenses; FSPO trains a continuous prefix against an evolving discrete attacker, coupling harmful refusal with benign preservation (Section 4, Eqs. 3–6).

  1. Defensive learning changes what the prompt must preserve. Earlier prompt-learning work learns task-dependent outputs—generated text or task labels—without simplex and entropy constraints. In jailbreak defense, harmful requests instead share a narrow refusal target while benign requests require varied, useful answers. Unconstrained optimization overfit refusal behavior or produce unnatural ordinary responses.

  2. Direct Z experiment motivates constraints for preserving benign utility. Since unconstrained optimization can overfit to refusal or degrade utility, we introduce simplex and entropy constraints to guide defensive learning toward preserving benign responses as well as refusal behavior.

  3. We adapt Geisler et al.'s (2024) simplex/entropy PGD from attack generation to defensive learning. Their method evaluates argmax-discretized candidates and returns discrete attack text; FSPO deploys its continuous defense. The contribution is this defensive formulation and its measured safety–utility and computational trade-offs.

  4. Section 5.1 realizes strict improvement over exact discrete optima. Equation (7) establishes feasible-set inclusion and a no-worse global optimum for the same loss. With one defensive position and three fixed attacks, exhaustive vocabulary enumeration identifies each best discrete token; FSPO achieves lower refusal loss in all three cases. The toy result, Soft/Hard study and new controls support constrained continuous defense as a promising direction.

Revision location: The revision adds the coefficient ablation in Table R3 (p. 9) and specifies the entropy target in Appendix E.2 (p. 18). Appendix G reports the coordinate control and benign utility in Table R1 (p. 22); please see Figure R2 (p. 22) for the embedding geometry and coefficient concentration. Section 5.1 (p. 5), Figure 2 (p. 6), and Figure 8 and Tables 12–13 (p. 21) provide the complementary toy and Soft/Hard evidence.

W2. Inherited safety alignment

FSPO adds substantial protection to the same frozen Vicuna backbone.

To address “by testing whether the gains disappear when the base model's safety behavior is removed,” we propose the closest practical proxy available in our experiments. Our proxy asks whether FSPO provides substantial protection when the starting model has weaker documented safety alignment. We do not believe a validated procedure currently exists for completely removing inherited safety while preserving other capabilities. Steering and safety unlearning suppress refusal (Arditi et al., 2024; Song et al., 2025), but suppression alone does not verify complete removal; unlearning evaluations can miss retained knowledge or collateral capability changes (Lynch et al., 2024). We welcome a protocol establishing both properties.

  • Vicuna-7B-v1.3 uses pretrained LLaMA-1 weights and ShareGPT supervised tuning, without separately documented RLHF (Zheng et al., 2023).

  • Llama-2-Chat uses different pretrained weights and safety SFT/RLHF (Touvron et al., 2023).

We compare the same Vicuna before/after FSPO, with Llama-2-Chat as the safety-aligned reference on identical inputs. This gives the reviewer's question a concrete experimental counterpart: a weaker-alignment starting point, the protection gained from FSPO, and an aligned reference evaluated on the same inputs.

New Vicuna-targeted GCG instances use 100 predetermined HarmBench requests, 500 steps, 512 candidates, top-k256, 20 suffix tokens and one restart. Llama-2 receives the same Vicuna-targeted inputs as a static-transfer reference.

Model / interventionCorrespondence to the reviewer's questionUnsafe /100
VicunaStarting-point proxy with weaker documented safety alignment86
Same Vicuna + FSPOSafety gain from adding a trained prefix to the same frozen backbone19
Llama-2-Chat referenceReference for the protection obtained in a safety-aligned model0

FSPO reduces unsafe responses by 67 percentage points. The gain persists on this model with weaker documented alignment, without updating its backbone weights. On the shared attack set, the trained prefix moves Vicuna substantially toward the safety-aligned reference, directly addressing whether useful protection can be gained outside strongly aligned backbones.

The 19-versus-0 comparison must be read in light of the attack target. Llama-2's 0/100 comes from replaying Vicuna-targeted attacks. In a separate evaluation, GCG optimized directly against undefended Llama-2 yields 8/100 Guard3-unsafe responses (100 steps, 128 candidates, top-k256). Thus, 0/100 is a transfer-attack result, not an attack-independent measure of its safety. The residual cross-model gap alone cannot determine whether FSPO learns a standalone defense: it compares different backbones and alignment histories.

FSPO achieves this gain with a small training budget. Our reported setting uses 25 harmful training requests plus 100 benign preservation prompts and approximately two hours on one A40, while keeping the backbone frozen. Llama-2-Chat's reference performance follows a broader alignment pipeline using 27,540 SFT annotations, including safety demonstrations, followed by safety-oriented RLHF (Touvron et al., 2023, Sections 3.1 and 4.2). FSPO thus adds substantial protection to a weaker-alignment backbone with a small training set and prefix-only training.

Vicuna's weaker results are not unique to FSPO. Original Table 1 gives static AutoDAN ASRs of 13% / 20% / 19% for FSPO/RPO/PAT on Vicuna, versus 0% / 0% / 0% on Llama-2-Chat. All three weaken on Vicuna; the new GCG experiment measures FSPO's gain on that backbone.

Revision location: Appendix G.1 (pp. 22–23) explains the safety-removal proxy, its interpretation and training-resource context. Table R11 (p. 23) reports the matched Vicuna comparison, including PAT/RPO/CB controls; original Table 1 (p. 6) gives the cross-model AutoDAN results.

Thank you for the response. The revised comparison with recent soft-prompt defenses remains model-limited. More fundamentally, FSPO essentially changes adversarial training from using discrete prompts to using continuous soft prompts, which I do not regard as sufficiently novel. I therefore keep my original rating unchanged.

Reviewer_M2XkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FSPO, a prompt-level jailbreak defense. It relaxes a defensive prefix into the vocabulary probability simplex and optimizes it with projected gradient descent, alternating with a GCG adversary that refines adversarial suffixes against the current defense. At deployment, the learned soft prompt is injected at the embedding layer. Experiments cover Llama-2-7B-Chat, Vicuna-7B and gpt-oss-20b, plus Qwen3.5-9B in the appendix. They test static transfer attacks (AdvBench→HarmBench) and defense-aware adaptive attacks (GCG, PAIR, AutoDAN). The paper reports lower adaptive ASR than RPO and PAT and a roughly 25× cheaper defense step. It also presents an attention analysis suggesting that the defensive prefix keeps more attention under attack.

Critical strengths

1.The defense is lightweight and efficient to optimize. FSPO trains only a short defensive prefix and avoids the candidate evaluations required by discrete prompt search. Table 3 reports a defense-phase iteration time of 3.5 seconds, compared with 91.5 seconds for PAT.

2.The results on Llama show a useful balance between robustness and utility. FSPO achieves an adaptive GCG ASR of 1%, compared with 7% for PAT (Table 2). It also retains better MT-Bench performance than PAT, scoring 5.37 versus 4.24 (Table 1).

Critical weaknesses
  1. The entropy projection is insufficiently motivated and specified. Algorithm 1 applies an entropy constraint that is absent from the optimization problem in Eq. (3), but the paper does not specify the target entropy or explain why this restriction is needed. Its effect on the search space, robustness, and benign utility remains unclear.

  2. The contribution relative to prior continuous prompt optimization is unclear. Soft prompt tuning (Lester et al., 2021) and PGD on continuously relaxed prompts (Geisler et al., 2024) already provide key components of the approach. Although these works are cited, the paper lacks a clear technical comparison and a standard soft-prompt baseline under the same adversarial training procedure, making the additional contribution difficult to assess.

  3. The evidence for generalization under adaptive attacks is incomplete. Table 2 includes training requests, while the cross-dataset HarmBench evaluation uses only static attacks. Figure 3(b) evaluates held-out behaviors but reports refusal loss rather than ASR, leaving adaptive robustness on unseen requests insufficiently validated.

Questions for authors
  1. Entropy projection. What target entropy or schedule is used, and why is this constraint needed for a prefix that is deployed continuously? Please clarify the corresponding optimization constraint and report how ASR and benign utility change when the projection is removed or its target is varied.

  2. Distinction from prior work. What is the specific methodological advance over combining existing continuous prompt optimization with adversarial training? Please compare FSPO with directly optimized prompt embeddings using the same training procedure, prompt length, and compute budget.

  3. Adaptive generalization. How many requests in Table 2 overlap with training, and what is the adaptive ASR on the disjoint subset? Please also report adaptive ASR on HarmBench using the same attack budgets.

3 comments

W1 / Q1. Why simplex and entropy constraints?

Refusal learning alone can sacrifice utility; simplex limits embedding excursions and entropy sharpens mixtures.

Section 4 constrains continuous prompts using Geisler et al.'s (2024) PGD. We test this motivation directly.

  1. Direct Z optimizes the defensive embedding matrix ZZ (20 vectors of dimension 4096), without simplex or entropy constraint. All 20 vectors leave the vocabulary convex hull: mean embedding L2 norm 39.17, versus a maximum of 1.27 across the model's original vocabulary embeddings, and mean hull distance bounded by [38.88, 39.41]. Direct Z produces 22/98 ASR, while ARC-C drops from 39.5% without a prefix to 16.5%; 14/25 benign responses contain malformed fragments. We hypothesize that off-hull embeddings contribute to this utility degradation.

  2. With frozen vocabulary embeddings EE, simplex coefficients AA give Z=AEZ=AE: nonnegative, unit-sum rows keep vectors inside the hull.

  3. Entropy sharpens coefficients toward one-hot rows (one selected token): mean nearest-one-hot distance falls 0.598→0.180, and mean largest coefficient rises 0.444→0.866. This concerns coefficient space: nearest-token embedding L2 instead rises 0.050→0.097. (Please see revision Figure R2, p. 22.) For coefficient row aa, Geisler et al.'s rule uses H2(a)=1−∥a∥22H_2(a)=1-\|a\|_2^2, our default factor η=0.8\eta=0.8, and nominal target (1−η)(k−1)/k(1-\eta)(k-1)/k, where kk is support size before sharpening. Appendix E.2 supplies the omitted target.

Direct Z updates embeddings ZZ directly; Off/Off updates unconstrained coefficients AA in Z=AEZ=AE, where EE is the frozen vocabulary-embedding matrix. Both can leave the convex hull, but their different optimization coordinates can yield different results.

The matched ablation fixes coefficient coordinates:

SimplexEntropy factorUnsafe /98ARC-C200
✗✗3224%
✓✗1831%
✓✓237%

In this ablation, adding simplex and then entropy improves both safety and utility.

Revision location: The revision adds the coefficient ablation in Table R3 (p. 9) and specifies the entropy target in Appendix E.2 (p. 18). Appendix G reports the coordinate control and benign utility in Table R1 (p. 22); please see Figure R2 (p. 22) for the embedding geometry and coefficient concentration.

W3 / Q3. Defense awareness versus training overlap

Both AdvBench subsets are attacked against the known deployed defense; training overlap is a separate axis.

  • Training uses AdvBench 1–25.

  • Static attacks optimize against the undefended backbone, then are frozen and replayed on the defense. Table 1 uses HarmBench requests unseen in defense training.

  • Adaptive attacks optimize through the known deployed defense: Table 2 uses seen AdvBench 1–25 and unseen 26–100.

  • PAIR/AutoDAN were absent from training.

Adaptive attackers know the deployed defense and optimize against the defended model. Thus the comparison does not hide the defense or reduce attacker knowledge to favor FSPO.

Our reporting did not clearly separate training overlap from defense awareness or provide subset ASRs.

Number of successful attacks ↓, split by overlap with training requests. Of the 100 evaluation requests, 25 were seen during FSPO/RPO/PAT training and 75 were unseen; the same ID-based split is applied to all defenses. Both subsets are attacked with defense-aware attacks, so the Seen/Unseen comparison measures generalization to unseen requests.

ModelAttackFSPO Seen (/25)FSPO Unseen (/75)RPO Seen (/25)RPO Unseen (/75)PAT Seen (/25)PAT Unseen (/75)
Llama-2-ChatGCG0171116
Llama-2-ChatPAIR006602
Llama-2-ChatAutoDAN100200
VicunaGCG81811331050
VicunaPAIR215725752575
VicunaAutoDAN381310213
GPT-OSSGCG020322
GPT-OSSPAIR311720
GPT-OSSAutoDAN2992800

Additional adaptive HarmBench evaluation

We added defense-aware adaptive GCG evaluation on HarmBench requests, which are not used in training. Attacks are optimized separately against each deployed defense.

ModelEvaluated requestsPAT unsafeRPO unsafeFSPO unsafe
Llama-2-Chat100184
GPT-OSS20250

The GPT-OSS evaluation is still running; we will report updated results as more requests are evaluated.

Revision location: Appendix A (pp. 14–15) describes the original protocol, and Table 2 (p. 7) reports pooled results. The revision adds the model-by-defense adaptive HarmBench table (Table R12, p. 23) and the seen/unseen split results (Table R13, p. 24).

W2 / Q2. Contribution beyond soft prompt tuning and PGD

Our contribution is constrained continuous adversarial training for jailbreak defense, with continuous prompts at both training and deployment.

We formulate prompt-tuning-based jailbreak defense as constrained continuous adversarial training, building on continuous prompt learning (Lester et al., 2021; Li & Liang, 2021) and adversarial training (Madry et al., 2017) for a frozen LLM. RPO/PAT optimize discrete defenses (Zhou et al., 2024; Mo et al., 2024); FSPO trains a continuous prefix against an evolving discrete attacker, coupling harmful refusal with benign preservation (Section 4, Eqs. 3–6).

  1. Defensive learning changes what the prompt must preserve. Earlier prompt-learning work learns task-dependent outputs—generated text in Li & Liang (2021), and task labels or answers in Lester et al. (2021)—without FSPO's vocabulary-simplex and entropy constraints. In jailbreak defense, harmful requests instead share a narrow refusal target while benign requests require varied, useful answers. Unconstrained optimization can overfit refusal behavior or produce unnatural ordinary responses.

  2. Direct Z experiment motivates constraints for preserving benign utility. Since unconstrained optimization can overfit to refusal or degrade utility, we introduce simplex and entropy constraints to guide defensive learning toward preserving benign responses as well as refusal behavior.

  3. We adapt Geisler et al.'s (2024) simplex/entropy PGD from attack generation to defensive learning. Their method evaluates argmax-discretized candidates and returns discrete attack text; FSPO deploys its continuous defense. We credit these inherited operations: their role here is regularizing harmful refusal plus benign preservation. The contribution is this defensive formulation and its measured safety–utility and computational trade-offs.

Section 5.1 realizes strict improvement over exact discrete optima.

Equation (7) establishes feasible-set inclusion and a no-worse global optimum for the same loss. With one defensive position and three fixed attacks, exhaustive vocabulary enumeration identifies each best discrete token; FSPO achieves lower refusal loss in all three cases (please see revision Figure 2, p. 6). The toy result, Soft/Hard study and new controls support constrained continuous defense as a promising direction.

A separate equal-data/length/150-update control gives Direct Z/simplex-only/FSPO 22/18/2 unsafe /98, 16.5/31.0/37% ARC-C. These controls examine continuous parameterizations and constraints; the Soft/Hard grid more directly examines continuity itself.

Revision location: Sections 2.2 and 4 describe the formulation; Equation (7) (p. 4), Section 5.1 (p. 5) and Figure 2 (p. 6) provide the relaxation argument and toy realization. Table 3 (p. 8), Figure 8 and Tables 12–13 (p. 21), and the new Table R1 (p. 22) give the computational and empirical comparisons.

Additional baselines use author artifacts where available, with explicit reconstruction and evaluation differences.

DefenseUnsafe /98
DDPO, author-code training11
DRO, released tensor12
MCD, local reconstruction15
FSPO2
  • DDPO uses unmodified author training code and all 600 released examples; evaluation aligns layer 25→26 and extends the generation cap 32→256.

  • DRO (Zheng et al., 2024) uses the unchanged released 116-vector tensor in its native system position.

  • MCD is a provisional local reconstruction: missing helpers, data, probes and configurations require different translation, probe-fitting and judging components.

Our GCG inputs provide a more challenging test than the baseline papers' evaluations.

Our budget is 500 steps, 256 candidates and top-k256 (Appendix B). DRO's released traces contain at most 100 logged iterations (Zheng et al., 2024); MCD evaluates multilingual harmful queries rather than GCG-optimized inputs (Li et al., 2025); DDPO uses a different GCG corpus whose search budget is not recoverable from the inspected paper/release (Obidov et al., 2026). Stronger attack prompts are a plausible contributor to the published-result gap.

Revision location: Appendix G describes the additional baseline implementations and evaluation differences, with their shared-input results in Table R2 (p. 22).

We thank the reviewer again for suggesting adaptive evaluation on HarmBench under matched attack budgets. We have now extended this evaluation to GPT-OSS-20B. As in our earlier response, we construct defense-aware adaptive GCG attacks on HarmBench requests unseen during training. The table below also includes the Llama-2-Chat results from our earlier response:

ModelEvaluated requestsPATRPOFSPO
Llama-2-Chat100184
GPT-OSS-20B503120
  • On GPT-OSS-20B, FSPO is the only defense with no successful adaptive attack , compared with 3/50 for PAT and 12/50 for RPO.
  • On Llama-2-Chat, FSPO ranks second (4/100).

This suggests its robustness is not tied to a single architecture.

We will include the complete 50-request GPT-OSS-20B results and the full adaptive-attack protocol in the revised version.

We would appreciate your feedback on whether these results, together with our earlier responses, address your concerns. As the rebuttal period draws to a close, please let us know if any concerns remain or need further clarification.

Reviewer_h219OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

FSPO defends against jailbreaks by optimizing a continuous soft prompt over the vocabulary simplex. Training alternates GCG attacks with defense updates that balance refusal and benign-response losses. Its geometric argument compares global optima, without claiming that lower training loss guarantees safer outputs. Tests across several models report lower adaptive attack success than RPO and PAT, along with faster defense updates. Utility is also measured, with MT-Bench falling from 6.16 without defense to 5.37 with FSPO.

Critical strengths
  • The geometric claim is carefully scoped: Equation 7 does not promise that training reaches a better solution or that lower loss means lower attack success. Figure 2 adds a small but direct comparison with the best discrete token.
  • Table 2 evaluates attacks that know the deployed defense. On Llama GCG, FSPO keeps attack success at 1%, compared with 18% for RPO under adaptive attack.
  • Table 3 directly measures the speed benefit: defense updates take 3.5 seconds versus 91.5 seconds for PAT. The utility tests also make the cost of the safety gains visible.
Critical weaknesses
  • Section 2.2 cites other prompt-tuning defenses, but the main prompt-level comparisons use only RPO and PAT. The differences from those other methods and the reasons for excluding them need clarification.
  • Training uses 25 harmful prompts, and most ASR tests contain about 100 prompts (Appendices A and G). Small percentage gaps therefore involve few examples, with no seed-level uncertainty reported despite acknowledged optimization noise.
  • Algorithm 3 uses a target entropy whose value is not listed in the setup. Appendix G does not isolate the entropy projection, leaving its effect on robustness and convergence unclear.
Questions for authors
  • How does FSPO differ from the other prompt-tuning defenses cited in Section 2.2? Which comparison best separates continuous relaxation from the rest of the training design?
  • How many independent runs support Tables 1 and 2? Could you report seed variation and confidence intervals for the smaller ASR gaps?
  • What entropy target was used in Algorithm 3? How do adaptive results change when the projection is removed or its target is varied?
3 comments

We thank the reviewer for their careful reading and constructive feedback, and address each point below.

W1 / Q1. Baseline scope and comparisons of continuous relaxation

RPO/PAT address the same adversarial prompt-defense problem with a frozen backbone; we now add continuous-defense baselines.

The original scope was adversarially optimized, prompt-based defense (Sections 2.2/5.2). The related work also includes manually designed safety prompts (Wu et al., 2023) and evolutionary system-message search (Zou et al., 2024b), which do not use the same prompt-training design. This explains the emphasis on RPO/PAT, but we agree that continuous prompt defenses are relevant comparisons and should also be included. We therefore add DDPO (Obidov et al., 2026), DRO (Zheng et al., 2024) and MCD (Li et al., 2025).

The methods differ in what is learned and where it acts:

MethodLearned intervention / objectiveRelation to FSPO
RPO / PATDiscrete defensive token prompts optimized against attacksPrimary comparison within adversarial prompt optimization
DDPOInput-dependent deep prompts inserted at an internal layerDifferent conditioning and injection location
DROContinuous system prompt optimized using refusal-direction representationsRelevant soft-prompt defense with a different objective
MCDMultilingual collaborative defense with learned promptsDifferent multilingual training and inference pipeline
FSPOContinuous input prefix, alternating adversarial training, simplex/entropy constraintsContinuous deployment with a refusal-plus-utility objective

We evaluate the same 98 Llama-2-Chat GCG inputs (500 steps/256 candidates/top-k256; Appendix B), with Guard3:

DefenseUnsafe /98
DDPO, author-code training11
DRO, released tensor12
MCD, local reconstruction15
FSPO2
  • DDPO uses unmodified author training code and all 600 released examples; evaluation aligns layer 25→26 and extends the generation cap 32→256.

  • DRO (Zheng et al., 2024) uses the unchanged released 116-vector tensor in its native system position.

  • MCD is a provisional local reconstruction: missing helpers, data, probes and configurations require different translation, probe-fitting and judging components.

Our GCG inputs provide a more challenging test than the baseline papers' evaluations.

Our budget is 500 steps, 256 candidates and top-k256 (Appendix B). DRO's released traces contain at most 100 logged iterations (Zheng et al., 2024); MCD evaluates multilingual harmful queries rather than GCG-optimized inputs (Li et al., 2025); DDPO uses a different GCG corpus whose search budget is not recoverable from the inspected paper/release (Obidov et al., 2026). Stronger attack prompts are a plausible contributor to the published-result gap.

Moreover, the Soft versus Hard optimization study in Appendix G (Figure 8 and Tables 12–13) provides a complementary comparison that more directly examines continuous relaxation itself.

It crosses continuous/discrete defense updates with attacker budgets and epochs, reporting defender update counts and costs. Comparisons at the same attacker-budget/epoch setting more directly examine representation choice.

The new controls examine which constraints make continuous defense useful: Direct Z versus simplex-only/FSPO compares parameterizations, while the coefficient-space ablation fixes optimization coordinates. Please see W3 / Q3 for this complementary evidence. Together, these comparisons provide complementary evidence about continuous representation and the constraint choices within it.

Revision location: Appendix G reports baseline implementations and results in Table R2 (p. 22), alongside the Soft/Hard study in Figure 8 and Tables 12–13 (p. 21). The new constraint and coordinate controls appear in Tables R3 (p. 9) and R1 (p. 22).


W2 / Q2. Independent runs and additional Vicuna evaluation

The original tables aggregate three seeds; the new repetitions quantify variation explicitly.

FSPO gives 2, 1, 2 unsafe /98.

Separately, we evaluate fixed checkpoints on 100 newly generated Vicuna-targeted GCG inputs, distinct from Table 1's attack instances (500 steps/512 candidates):

Model / defenseUnsafe /100
Vicuna86
Vicuna+FSPO19
Vicuna+PAT24
Vicuna+RPO28
Vicuna+CB68

FSPO improves over undefended Vicuna and has lower unsafe counts than PAT/RPO on this new static set.

Revision location: The revised Section 5.2 (p. 7) reports the three-run unsafe counts. Appendix G.1 (pp. 22–23) explains the weaker-alignment comparison, with the Vicuna results and PAT/RPO/CB controls in Table R11 (p. 23).

W3 / Q3. Why simplex and entropy constraints?

Simplex bounds embedding excursions; entropy sharpens mixtures inside that hull.

  1. Direct Z directly optimizes the defensive embedding matrix ZZ (20 vectors, dimension 4096), without either constraint. All 20 vectors leave the vocabulary hull: mean embedding L2 norm 39.17, versus a maximum of 1.27 across the model's original vocabulary embeddings, and mean hull distance bounded by [38.88, 39.41]. Direct Z produces 22/98 Guard3-unsafe responses, while ARC-C drops from 39.5% without a prefix to 16.5%, with malformed fragments in 14/25 benign responses. We hypothesize that off-hull embeddings contribute to this utility degradation.

  2. Simplex restricts search to pretrained embedding mixtures. Let EE be the frozen vocabulary-embedding matrix and AA the mixing coefficients. With Z=AEZ=AE, nonnegative unit-sum rows of AA confine each vector to the hull.

  3. Entropy sharpens mixtures inside that hull, pushing them toward one-hot distributions, where one coefficient selects one token. In the matched pair, mean distance to the nearest one-hot row decreases 0.598→0.180, and the mean largest coefficient increases 0.444→0.866. For a coefficient row aa, we use H2(a)=1−∥a∥22H_2(a)=1-\|a\|_2^2, fixed factor η=0.8\eta=0.8, and nominal target (1−η)(k−1)/k(1-\eta)(k-1)/k, where kk is support size before sharpening.

Direct Z updates embeddings ZZ directly; simplex/entropy Off/Off updates unconstrained coefficients AA in Z=AEZ=AE, where EE is the frozen vocabulary-embedding matrix. Both can leave the convex hull, but their different optimization coordinates can yield different results.

The controlled ablation fixes coefficient coordinates:

SimplexEntropy factorUnsafe /98ARC-C200
✗✗3224%
✓✗1831%
✓✓237%

In this ablation, adding simplex and then entropy improves both safety and utility.

Our contribution is constrained continuous adversarial training for jailbreak defense, with continuous prompts at both training and deployment.

We formulate prompt-tuning-based jailbreak defense as constrained continuous adversarial training, building on continuous prompt learning (Lester et al., 2021; Li & Liang, 2021) and adversarial training (Madry et al., 2017) for a frozen LLM. RPO/PAT optimize discrete defenses (Zhou et al., 2024; Mo et al., 2024); FSPO trains a continuous prefix against an evolving discrete attacker, coupling harmful refusal with benign preservation (Section 4, Eqs. 3–6).

  • Defensive learning changes what the prompt must preserve. Earlier prompt-learning work learns task-dependent outputs—generated text in Li & Liang (2021), and task labels or answers in Lester et al. (2021)—without FSPO's vocabulary-simplex and entropy constraints. In jailbreak defense, harmful requests instead share a narrow refusal target while benign requests require varied, useful answers. Unconstrained optimization can overfit refusal behavior or produce unnatural ordinary responses.

  • Direct Z experiment motivates constraints for preserving benign utility. Since unconstrained optimization can overfit to refusal or degrade utility, we introduce simplex and entropy constraints to guide defensive learning toward preserving benign responses as well as refusal behavior.

  • We adapt Geisler et al.'s (2024) simplex/entropy PGD from attack generation to defensive learning. Their method evaluates argmax-discretized candidates and returns discrete attack text; FSPO deploys its continuous defense. We credit these inherited operations: their role here is regularizing harmful refusal plus benign preservation. The contribution is this defensive formulation and its measured safety–utility and computational trade-offs.

  • Section 5.1 realizes strict improvement over exact discrete optima. Equation (7) establishes feasible-set inclusion and a no-worse global optimum for the same loss. With one defensive position and three fixed attacks, exhaustive vocabulary enumeration identifies each best discrete token; FSPO achieves lower refusal loss in all three cases (please see revision Figure 2, p. 6). The toy result, Soft/Hard study and new controls support constrained continuous defense as a promising direction.

Revision location: The revision adds the coefficient ablation in Table R3 (p. 9) and specifies the entropy target in Appendix E.2 (p. 18). Appendix G reports the coordinate control and benign utility in Table R1 (p. 22); please see Figure R2 (p. 22) for the embedding geometry and coefficient concentration. Section 5.1 (p. 5), Figure 2 (p. 6), and Figure 8 and Tables 12–13 (p. 21) provide the complementary toy and Soft/Hard evidence.

Dear Reviewer, thank you for your feedback. As the discussion period draws to a close, could you let us know whether our responses have addressed your concerns? We would appreciate hearing if any concerns remain or need further clarification.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Clarifying FSPO’s Contribution Through Additional Evidence

We formulate prompt-tuning-based jailbreak defense as constrained continuous adversarial training and provide evidence supporting this formulation. Our contribution goes beyond combining continuous prompt learning, adversarial training, and PGD: Section 4 (Eqs. 3–6) formalizes the defensive objective, and Section 5.1 motivates continuous relaxation [1]. Our new Direct Z controls and simplex/entropy ablations establish the empirical rationale for constraints: continuous flexibility must serve both harmful refusal and benign-response preservation [2, 3].

  • [1] Continuous relaxation — original rationale. Section 5.1/Figure 2 realizes lower refusal loss than exact discrete optima in the controlled case study with exhaustive vocabulary search; the Soft/Hard study further examines representation choice. These motivate continuous defense as a promising direction. Rebuttal: M2Xk W2; h219 W1.

  • [2] Defensive constraints — new Direct Z evidence. Direct Z directly optimizes unconstrained prompt embeddings. Our added utility probes and geometry audit reveal departure from the vocabulary hull alongside degraded, sometimes malformed benign responses. This exposes a failure mode of unconstrained refusal fitting and motivates constraints tailored to defensive learning. Rebuttal: h219 W3; M2Xk W1; yhKh W1.

  • [3] Constraint design — new controlled ablation. Holding coefficient coordinates fixed, adding simplex and then entropy improves both safety and benign utility. Geometry measurements document hull containment and sharpening toward one-hot coefficients. This supports adapting Geisler et al.'s (2024) attack-oriented PGD to defensive learning. Rebuttal: h219 W3; M2Xk W1–W2; yhKh W1.

  • [4] Practical scope — new comparative evidence. Added DDPO/DRO/MCD comparisons show competitive defense, while matched Vicuna controls demonstrate gains on a weaker-aligned backbone. Rebuttal: baselines—h219 W1, M2Xk W2, yhKh W1; Vicuna—yhKh W2.

Revision additions are marked in red; original text is preserved.

Area_Chair_abRSOpenReview ↗

Reminder: Please review the author rebuttal before the deadline

Dear reviewers,

The rebuttal period is about to close. If you haven’t responded yet, please read the authors’ rebuttal and leave a brief comment before the deadline.

If you’ve already responded, please disregard this reminder. Thanks for your time, and apologies for the extra email.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.