- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how jailbreak attack success rates scale with the number of independently generated responses per prompt. Starting from the distribution of prompt-conditioned single-response safety probabilities, the authors derive an expression for attack success under repeated sampling and propose a scaling form combining polynomial and exponential factors. A spin-glass model is further introduced to provide a theoretical interpretation of the scaling behavior under different attack strengths. The empirical evaluation covers GCG, AutoDAN, and no-injection settings across multiple language models and two benchmarks, AdvBench and HarmBench, together with comparisons of evaluation methods and attack-category analyses. The results show that prompt injection increases attack success within the evaluated settings and that the proposed scaling form provides good fits to the observed curves.
Critical strengths
S1. Broad empirical evaluation with complementary judge validation
The paper provides a reasonably broad empirical evaluation. Figures 3–4 and Appendix H cover multiple language models, two attack methods (GCG and AutoDAN), two benchmarks (AdvBench and HarmBench), and category-level analyses, offering evidence across different settings that prompt injection alters attack-success curves under finite sampling budgets. The authors also address potential evaluation bias arising from refusal-string-based metrics. Specifically, Appendix G.2 compares refusal-based and semantic evaluations across different sampling budgets, while Appendix G.4 examines agreement between two LLM-based judges. These complementary evaluations strengthen the credibility of the empirical findings beyond what could be established by a single model, attack configuration, or evaluation metric.
S2. An informative probabilistic framework supported by theoretical-model simulations
The paper develops an informative probabilistic framework for analyzing jailbreak success under repeated sampling. By expressing the probability of no successful jailbreak after k independent generations as E[Pᵏ], the analysis connects the scaling behavior to heterogeneity across prompt-conditioned safety probabilities and the upper-tail properties of their distribution. This offers a richer perspective than evaluating single-response attack success rates alone. In addition, the numerical simulations in Appendix F, particularly the comparison with analytical predictions in the weak-field regime, provide supporting evidence for the behavior predicted within the theoretical model. Although the connection between the spin-glass model and actual LLM behavior remains primarily qualitative, the proposed framework offers a valuable theoretical perspective on repeated-sampling jailbreak dynamics.
Critical weaknesses
W1. Insufficient evidence for the necessity and robustness of the exponential term
A central empirical claim of the paper relies on the proposed polynomial–exponential scaling form, yet the current analysis does not sufficiently establish the necessity or robustness of the exponential component. Appendix G.5.3 fits the transformed quantity log(−log Πₖ) using ordinary least squares and reports high R² values. However, goodness of fit alone does not demonstrate that the model with an exponential component (μ > 0) provides a statistically or predictively meaningful improvement over a pure power-law model (μ = 0), nor does it establish that the observed finite-budget trends represent an asymptotic polynomial-to-exponential crossover. The paper does not report direct comparisons against the simpler model, uncertainty estimates for the key parameters μ and ν, or out-of-range predictive validation using larger sampling budgets. Although Appendix G.5.1 reports two independent generation sets and corresponding ASR statistics, these do not directly quantify uncertainty in the fitted scaling parameters.
Furthermore, the caption of Figure 3 acknowledges a numerical limitation around 1 − Πₖ ≈ 10⁻³ for high-injection curves. As ASR approaches one, the fitted transformation may become particularly sensitive to estimation errors, potentially affecting the reliability of tail behavior and parameter estimates. The manuscript does not clearly specify whether these boundary-affected points are included in the fits or whether the central conclusions remain stable under alternative treatments. While this does not imply that the exponential component is a numerical artifact, its sensitivity to these limitations remains insufficiently examined. Direct comparisons between the pure power-law and mixed models, together with parameter uncertainty estimates and boundary-sensitivity analyses, would substantially strengthen the evidence for the claimed crossover. (pp. 9, 65–66)
W2. Limited empirical support for the proposed distributional assumptions and mechanistic interpretations
The empirical evidence connecting the theoretical assumptions to actual LLM behavior remains limited. First, the proposed exponential decay is associated with an upper endpoint strictly below one in the distribution of prompt-conditioned single-response safety probabilities. Although the finite-sample histograms in Figure 1 are qualitatively consistent with this assumption, they do not establish the essential supremum of the underlying distribution or directly validate its near-endpoint tail behavior. Moreover, the observed finite-budget scaling curves may be influenced by the discrete nature of the evaluated prompt set and uncertainty in estimated single-response safety probabilities. Consequently, obtaining good fits to the proposed scaling form is distinct from empirically validating the distributional assumptions required by the theoretical analysis. (p. 2)
Second, Section 5 relates the fitted parameters to reasoning depth and adversarial order, while using injection length as a proxy for effective field strength. Although Figure 2 provides some supporting trends, the evaluation includes only three injection-length settings, does not report parameter uncertainty, and does not exhibit a consistently monotonic trend in ν across all models. More importantly, the paper does not independently measure reasoning depth or reasoning ability, making it difficult to determine whether changes in the fitted parameters genuinely correspond to changes in these properties. The results therefore support a qualitative correspondence between the theoretical framework and empirical observations, but fall short of establishing the proposed mechanistic relationships. The authors should more clearly distinguish theoretical interpretations from experimentally validated conclusions and qualify the corresponding mechanistic claims accordingly. (pp. 7–8)
Questions for authors
See Weeknesses
We are very much thankful to the reviewer for the time and the precise set of questions. We address the concerns with new analyses below.
How the floor is chosen. We thank the reviewer for pressing on this. With M=520 prompts in AdvBench, 1−Π^k is quantized in steps of 1/M=1.9×10−3. This is why our code clips 1−Π^k at 10−3 where finite size effects become important and log(−log(1−1/520))=−6.25 is what determines the plateau in Fig. 3, 4. We have added this discussion to Fig. 3 caption in the revised version of the paper.
Uncertainty on ν^,μ^. In the paper, the goodness of the fit is measured by R2, which is calculated only from points that are above the floor value. We agree with the reviewer that this does not tell us about uncertainty in the fitted parameters in repeated trials. In the plots, we have already reported variance from repeatedly sampling the same LLM at a given temperature. In addition, we now bootstrap the entire pipeline over prompts resampling (2000 times) the 520 AdvBench prompts with replacement, recomputing Πk):
We have now added a detailed section on statistical measures of the fitted parameters in Appendix I.2 (confidence interval measure) and Appendix I.3 (nested comparison, μ=0 vs μ>0, see our reply to reviewer 9BKQ) in the revised paper and refer them in section 5 of main text.
What a finite-sized experiment can and cannot establish. We totally agree that no finite-k experiment can establish an asymptotic crossover, and we now state the limitation quantitatively. Let F be any law of the per-prompt safe probability P with p^=e−μ^<1 satisfying Assumption 1, and write gF(k)=EF[Pk]=1−Πk. Add a fraction w of prompts with p^=1: Gw=(1−w)F+wQν with P(Qν≥1−ε)=εν. Then: let k∗(w):=min(k≥1: wgQ(k)≥gF(k)), with gQ(k)=E[Qνk], be the first k at which the added component overtakes the original curve. Because gQ(k)≤1, this can happen only after the original curve has fallen below w: k∗(w)≥min{k:gF(k)≤w}. gF(k) sees a finite size effect when it is of order 1/M, and hence w<1/M is not ruled out by our experiments. We now state the limitation quantitatively in Appendix I.1 of the revised paper.
μ, ν as a function of attack strength. The theory predicts monotone growth of ν only in the weak-field regime (Remark 2) - not in general. The corresponding LLM Fig. 2 also shows the growth of ν^ in the small jailbreak strength regime - so the later non-monotonicity is consistent in the theoretical model. In the strong-field regime, the leading-order theory fixes the exponential rate and as the magnetic field increases, that increases as shown in Appendix B, and says nothing about the pre-factor exponent. In the corresponding LLM Fig. 2, we see that at large jailbreak strength μ^ does increase significantly. However, we agree with the reviewer on the criticism that we have not performed an extensive amount of experiments focused on understanding these - in particular, one can perform the same experiment at different points of GCG optimization strings and use it as the proxy of variable jailbreak strength. Also, we agree that the sentences interpreting ν^ as "lack of reasoning" following the Souly et al. capability observation are speculative, and we have moved this speculation to the caption of Fig. F.1.
SpinLLM vs realistic LLM. Please see our response to see our reply to reviewer j2x2.
Thank you so much for the review, please let us know if there are more questions.