- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the characterization and efficient discovery of bijection-based jailbreak attacks against large language models (LLMs). The authors argue that the conventional dispersion parameter is insufficient to characterize attack effectiveness because it does not account for how a particular character substitution interacts with the input prompt. To address this limitation, they introduce Prompt-Level Perturbation (PLP), which measures the proportion of prompt characters modified by a bijection.
The paper presents empirical evidence that attack outcomes exhibit structured patterns across different PLP levels and that favorable perturbation levels vary across prompts. Building on these observations, the authors propose Signed Langevin Search (SLS), a gradient-free stochastic search strategy that uses model feedback to navigate the perturbation space.
Experiments across multiple LLMs, including open-weight and proprietary models, and three jailbreak benchmarks demonstrate improvements in queries-to-first-attack compared with random bijection selection. The paper aims to provide both a more informative characterization of bijection attacks and a more query-efficient attack discovery method.
Critical strengths
S1. Prompt-Specific Characterization of Bijection Attacks. The introduction of Prompt-Level Perturbation provides a useful perspective for analyzing bijection attacks beyond the conventional dispersion parameter. By explicitly considering the interaction between a character mapping and its target prompt, the proposed measure captures variations that dispersion alone cannot distinguish. This characterization helps explain why bijections with identical dispersion values can produce different attack outcomes.
S2. Empirical Identification of Structured Attack Regimes. The paper provides interesting empirical observations suggesting that refusal, successful jailbreaks, and incoherent responses occupy different regions of the perturbation space. The analysis of prompt-dependent optimal perturbation levels further motivates adaptive attack strategies. These findings contribute to understanding the relationship between encoded input transformations and LLM safety behavior.
Critical weaknesses
W1. Limited Methodological Novelty of the Proposed PLP Measure. The proposed Prompt-Level Perturbation is essentially the fraction of input characters affected by the bijection, as defined in Equation (2). Although this measure is more informative than dispersion, it is closely related to a straightforward character-level modification ratio. The paper does not sufficiently establish why this relatively simple statistic constitutes a substantial methodological contribution. Furthermore, PLP ignores the semantic importance, positional distribution, and tokenization effects of substituted characters. Different bijections with identical PLP values may therefore induce substantially different model behaviors. More systematic comparisons with alternative perturbation measures are needed to establish the distinctive value of PLP.
W2. Insufficient Theoretical Justification for Signed Langevin Search. The proposed SLS algorithm is motivated by Langevin dynamics, but its actual update rule replaces the gradient of the log-success probability with a heuristic direction inferred from refusal and incoherence labels. The assumption that refusals consistently indicate the need for larger perturbations, while incoherent responses indicate the need for smaller perturbations, is not theoretically established. Moreover, the introduction of independently tuned noise, momentum, and annealing further separates the algorithm from conventional Langevin dynamics. Consequently, it remains unclear whether the observed improvements arise from the Langevin-inspired formulation or simply from directional heuristic search. This weakens the methodological justification for the central algorithmic contribution.
Questions for authors
Q1. Distinctive Value of PLP. How does PLP compare with other simple perturbation measures, such as character-level edit distance, tokenization change rate, or frequency-weighted substitution ratios? Can the authors provide quantitative evidence that PLP offers additional predictive value beyond these alternatives?
Q2. Validity of the SLS Search Direction. What empirical evidence supports the assumption that refusal and incoherence reliably indicate opposite directions toward higher attack success probability? How frequently does this assumption fail across different prompts and models, and how sensitive is SLS to such violations?
Q3. Contribution of the Langevin-Inspired Components. Can the authors compare SLS against simpler PLP-guided search strategies, including deterministic directional search and adaptive interval search, under identical candidate sets and query budgets? Such comparisons would help establish whether stochastic noise, momentum, and annealing contribute meaningfully to the reported improvements.
Q4. Reliability of Response Classification. How robust are the experimental results to errors in the GPT-4o-mini judge and the keyword-based refusal classifier? Have the authors evaluated classification agreement against human annotations or independent safety classifiers, particularly for ambiguous responses?
Q5. Interpretation of Query Efficiency. Since prompts without successful candidates are excluded from QTFA evaluation, how would the conclusions change when all prompts are included and unsuccessful attempts are treated as censored observations? Can the authors additionally report attack success rates under fixed query budgets?
Q6. Statistical Reliability of Prompt-Specific Optimal PLP. In Section 3.3, the optimal PLP for each prompt is estimated using 100 attacks. How stable are these estimated optima across independent sampling runs? Could the observed variation between prompt-specific and global optima partly result from finite-sample estimation noise?
6 comments
Q4 Response Classification
In this work, we use the exact classifier from the original bijection work (Huang et al. 2025), GPT4-mini using the popular Harmbench prompt. To further validate our analysis, we re-judge our Harmbench and Advbench results using the two open-source HarmBench classifiers (Llama-2-13B and Mistral-7B; Mazeika et al., 2024) which have been validated against humans. Here, using the same scheme as Q2 (averaged across all models), ordering remains consistent across the different judges. Further, average AUC values (across the 3 different classification tasks) indicate similar separability among these measures. The updated experiment is in Appendix G.1.
| Success judge | % Ordered [95% CI] | Avg. AUC [95% CI] |
|---|---|---|
| HarmBench | ||
| GPT-4o-mini | 71.2 [69.2, 73.4] | 0.630 [0.623, 0.638] |
| HarmBench classifier (Llama-2-13B) | 74.5 [72.2, 77.0] | 0.650 [0.641, 0.659] |
| HarmBench classifier (Mistral-7B) | 74.5 [72.2, 76.9] | 0.649 [0.640, 0.658] |
| AdvBench | ||
| GPT-4o-mini | 74.1 [68.8, 78.9] | 0.672 [0.654, 0.691] |
| HarmBench classifier (Llama-2-13B) | 82.2 [76.4, 87.4] | 0.692 [0.666, 0.717] |
| HarmBench classifier (Mistral-7B) | 75.0 [69.0, 81.0] | 0.693 [0.671, 0.714] |
We validate the keyword-based refusal-incoherence judge in Appendix G.2/G.3. Here we demonstrate strong recall against existing list-based judges (see our response to Reviewer uNUM, Q2). Further, using an LLM to judge into 3 groups instead of our rule based method makes no significant difference in ordering or separability (Appendix G.3).
Q5 Query Efficiency
Including prompts that have no successes in the candidate set would not change any of the conclusions of this work. Since both SLS and random search draw from the same fixed pool of N = 20 candidate bijections per prompt, at budget 20, the prompt is attacked successfully by SLS if and only if it is attacked successfully by random search. The ASR is identical for both methods (Table 6). For, these prompts where no such successful attack exists, any search method will search the entire candidate space and fail to find a successful attack. Therefore, any penalty will apply to every search method and have no effect on relative ordering. The exact values can be computed as a direct result from Tables 1, 3, 4, and 6. We include ASR@K results in Appendix C.8.
Q6 Statistical Reliability
We have replaced Figure 4 with a version using HarmBench (320 prompts, 5 independent runs of 20 attacks each), which gives a larger sample size. To verify that the spread observed in Fig. 4a is not sampling noise, we run ANOVA and Kruskal-Wallace tests. Both tests yielded , supporting rejection of the null hypothesis of a single universal and confirming the variability across prompts. These tests also achieve for Advbench.
Thanks for the response. I would like to keep my score.
Thank you for your review and your questions. We answer them below.
We arrived at PLP after examining a large array of other metrics and concluded that PLP, despite its simplicity, is the metric that best captures the three outcome regimes of success, incoherence, and refusals. While we had not included this analysis in the original draft, we will incorporate it into the revision to further strengthen our argument that PLP is a conceptually simple, easy-to-compute, black-box metric that requires no knowledge of the model's internal architecture, weights, or tokenizer.
We measure the average AUC of threshold classifiers on each measure for the 3 classification tasks (refusal vs success, success vs incoherence, refusal vs incoherence). Here we observe that no semantic, positional, or frequency-weighted metric we tested outperforms PLP. Only token-level edit distance did, by ~0.003 AUC, but token-level metrics cannot be used in black-box settings. Please see Appendix F of our revision for full details.
Following your suggestion, we systematically analyzed the three-regime ordering with respect to PLP (ρ). Our ordering metric is defined as follows. We use threshold classifiers to separate refusal and success (τRS), and success and incoherence (τSI). A prompt is ordered if τRS<τSI. On Harmbench (20 trials per prompt), for Llama 3 8B, we find that 83% of prompts are ordered. We have included this experiment in Appendix B.4.
Our ablation over the Langevin components is in Appendix D. In Appendix D.1 we isolate and justify the role of stochasticity in the update. When σ=0 (deterministic directional search), SLS reduces to a sign based-update which impacts mean QTFA (increases from 5.78 to 6.34). This demonstrates the importance of the Langevin component. Appendix D.3 and D.4 demonstrate that adding momentum and noise annealing reduces both mean and median QTFA on Qwen3 4B and yields similar QTFA with and without momentum on Llama 3 8B and Gemma E4B. Finally, in Appendix C.6 we demonstrate that SLS outperforms a more simplistic median-outward search. We have updated our text to make the reference to these appendices more explicit.