- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Fast Soft Prompt Optimization (FSPO), a prompt-level jailbreak defense that optimizes a continuous defensive prefix in the vocabulary simplex against an adaptive adversarial suffix. The method is motivated by the cost of existing defenses and by their tendency to be circumvented by adaptive attacks. The authors evaluate FSPO under static and adaptive attack settings and report improved robustness with lower optimization cost than GCG-style baselines.
Critical strengths
- Clean continuous-relaxation formulation that avoids expensive token-level search.
- Min-min optimization explicitly accounts for an adaptive attacker.
- Reported per-step speedup and a deployable pre-installed prefix.
Critical weaknesses
-
The comparison baseline is outdated and omits the most relevant recent defenses. The main prompt-level baselines are RPO and PAT while more recent and closely related work is neither compared nor discussed: DDPO [1] (AAAI 2026) learns an input-dependent deep prompt in a frozen LLM, and MCD [2] (EMNLP Findings 2025) optimizes a continuous soft safety prompt. Since FSPO's central claim is continuous soft-prompt optimization, the absence of these comparisons makes it difficult to assess its novelty and relative effectiveness. At minimum, the paper should discuss these methods and, where feasible, include them as baselines.
-
The method depends on the base model's pre-existing safety alignment. FSPO performs well on aligned models such as Llama-2-7B-Chat and gpt-oss-20b, but remains much weaker on the unaligned Vicuna-7B-v1.3: static PAIR ASR is 40% (vs. 0% on Llama) and adaptive PAIR ASR is 78% (vs. 0% on Llama). The paper attributes this to FSPO "recover[ing] and reinforce[ing] the safety behaviors of the base model," which suggests that the method is not learning a standalone defense but is largely reactivating existing refusal capacity. This limits its applicability to weakly aligned or unaligned models and should be analyzed more directly, for example by testing whether the gains disappear when the base model's safety behavior is removed.
[1] Obidov et al. Dynamic Deep Prompt Optimization for Defending Against Jailbreak Attacks on LLMs. AAAI 2026
[2] Li et al. Multilingual Collaborative Defense for Large Language Models. Findings of EMNLP 2025.
Questions for authors
See weaknesses
3 comments
W2. Inherited safety alignment
FSPO adds substantial protection to the same frozen Vicuna backbone.
To address “by testing whether the gains disappear when the base model's safety behavior is removed,” we propose the closest practical proxy available in our experiments. Our proxy asks whether FSPO provides substantial protection when the starting model has weaker documented safety alignment. We do not believe a validated procedure currently exists for completely removing inherited safety while preserving other capabilities. Steering and safety unlearning suppress refusal (Arditi et al., 2024; Song et al., 2025), but suppression alone does not verify complete removal; unlearning evaluations can miss retained knowledge or collateral capability changes (Lynch et al., 2024). We welcome a protocol establishing both properties.
-
Vicuna-7B-v1.3 uses pretrained LLaMA-1 weights and ShareGPT supervised tuning, without separately documented RLHF (Zheng et al., 2023).
-
Llama-2-Chat uses different pretrained weights and safety SFT/RLHF (Touvron et al., 2023).
We compare the same Vicuna before/after FSPO, with Llama-2-Chat as the safety-aligned reference on identical inputs. This gives the reviewer's question a concrete experimental counterpart: a weaker-alignment starting point, the protection gained from FSPO, and an aligned reference evaluated on the same inputs.
New Vicuna-targeted GCG instances use 100 predetermined HarmBench requests, 500 steps, 512 candidates, top-k256, 20 suffix tokens and one restart. Llama-2 receives the same Vicuna-targeted inputs as a static-transfer reference.
| Model / intervention | Correspondence to the reviewer's question | Unsafe /100 |
|---|---|---|
| Vicuna | Starting-point proxy with weaker documented safety alignment | 86 |
| Same Vicuna + FSPO | Safety gain from adding a trained prefix to the same frozen backbone | 19 |
| Llama-2-Chat reference | Reference for the protection obtained in a safety-aligned model | 0 |
FSPO reduces unsafe responses by 67 percentage points. The gain persists on this model with weaker documented alignment, without updating its backbone weights. On the shared attack set, the trained prefix moves Vicuna substantially toward the safety-aligned reference, directly addressing whether useful protection can be gained outside strongly aligned backbones.
The 19-versus-0 comparison must be read in light of the attack target. Llama-2's 0/100 comes from replaying Vicuna-targeted attacks. In a separate evaluation, GCG optimized directly against undefended Llama-2 yields 8/100 Guard3-unsafe responses (100 steps, 128 candidates, top-k256). Thus, 0/100 is a transfer-attack result, not an attack-independent measure of its safety. The residual cross-model gap alone cannot determine whether FSPO learns a standalone defense: it compares different backbones and alignment histories.
FSPO achieves this gain with a small training budget. Our reported setting uses 25 harmful training requests plus 100 benign preservation prompts and approximately two hours on one A40, while keeping the backbone frozen. Llama-2-Chat's reference performance follows a broader alignment pipeline using 27,540 SFT annotations, including safety demonstrations, followed by safety-oriented RLHF (Touvron et al., 2023, Sections 3.1 and 4.2). FSPO thus adds substantial protection to a weaker-alignment backbone with a small training set and prefix-only training.
Vicuna's weaker results are not unique to FSPO. Original Table 1 gives static AutoDAN ASRs of 13% / 20% / 19% for FSPO/RPO/PAT on Vicuna, versus 0% / 0% / 0% on Llama-2-Chat. All three weaken on Vicuna; the new GCG experiment measures FSPO's gain on that backbone.
Revision location: Appendix G.1 (pp. 22–23) explains the safety-removal proxy, its interpretation and training-resource context. Table R11 (p. 23) reports the matched Vicuna comparison, including PAT/RPO/CB controls; original Table 1 (p. 6) gives the cross-model AutoDAN results.
Thank you for the response. The revised comparison with recent soft-prompt defenses remains model-limited. More fundamentally, FSPO essentially changes adversarial training from using discrete prompts to using continuous soft prompts, which I do not regard as sufficiently novel. I therefore keep my original rating unchanged.
W1. Continuous-defense baselines and constraint motivation
FSPO has lower static ASR than DDPO, DRO and our MCD reconstruction.
DDPO (Obidov et al., 2026) learns input-dependent deep prompts; MCD (Li et al., 2025) uses multilingual collaboration. FSPO learns a continuous prefix (Sections 2.2/5.2).
We evaluate 98 Llama-2-Chat GCG inputs:
DDPO uses unmodified author training code.
DRO (Zheng et al., 2024) uses the unchanged released 116-vector tensor.
MCD is a provisional local reconstruction: missing helpers, data, probes and configurations require different translation, probe-fitting and judging components.
Our GCG inputs provide a more challenging test than the baseline papers' evaluations. Our budget is 500 steps, 256 candidates and top-k256. DRO's released traces contain at most 100 logged iterations; MCD evaluates multilingual harmful queries rather than GCG inputs; DDPO uses a different GCG corpus whose search budget is not recoverable from the inspected paper/release. Stronger attack prompts are a plausible contributor to the published-result gap.
Simplex bounds embedding excursions; entropy sharpens mixtures inside the hull.
Direct Z learns the defense embedding matrix Z (20 vectors, dimension 4096), without simplex or entropy constraints. All 20 vectors lie far outside the vocabulary hull. Direct Z produces 22/98 unsafe responses, while ARC-C drops from 39.5% without a prefix to 16.5%, with malformed fragments in 14/25 benign responses. We hypothesize that off-hull embeddings contribute to this utility degradation.
Let E be the frozen vocabulary-embedding matrix and A the mixing coefficients. With Z=AE, nonnegative unit-sum rows keep each vector in the vocabulary hull.
Entropy sharpens toward one-hot rows, where one coefficient selects one token: mean nearest-one-hot distance falls 0.598→0.180, and the mean largest coefficient rises 0.444→0.866. Sharpening uses fixed factor 0.8.
The ablation fixes coefficient coordinates:
In this ablation, adding simplex and then entropy improves both safety and utility.
Our contribution is constrained continuous adversarial training for jailbreak defense.
We formulate prompt-tuning-based jailbreak defense as constrained continuous adversarial training, building on continuous prompt learning (Lester et al., 2021; Li & Liang, 2021) and adversarial training (Madry et al., 2017). RPO/PAT optimize discrete defenses; FSPO trains a continuous prefix against an evolving discrete attacker, coupling harmful refusal with benign preservation (Section 4, Eqs. 3–6).
Defensive learning changes what the prompt must preserve. Earlier prompt-learning work learns task-dependent outputs—generated text or task labels—without simplex and entropy constraints. In jailbreak defense, harmful requests instead share a narrow refusal target while benign requests require varied, useful answers. Unconstrained optimization overfit refusal behavior or produce unnatural ordinary responses.
Direct Z experiment motivates constraints for preserving benign utility. Since unconstrained optimization can overfit to refusal or degrade utility, we introduce simplex and entropy constraints to guide defensive learning toward preserving benign responses as well as refusal behavior.
We adapt Geisler et al.'s (2024) simplex/entropy PGD from attack generation to defensive learning. Their method evaluates argmax-discretized candidates and returns discrete attack text; FSPO deploys its continuous defense. The contribution is this defensive formulation and its measured safety–utility and computational trade-offs.
Section 5.1 realizes strict improvement over exact discrete optima. Equation (7) establishes feasible-set inclusion and a no-worse global optimum for the same loss. With one defensive position and three fixed attacks, exhaustive vocabulary enumeration identifies each best discrete token; FSPO achieves lower refusal loss in all three cases. The toy result, Soft/Hard study and new controls support constrained continuous defense as a promising direction.