acceptodds
Under review as a conference paper at ICLR 2027

Jailbreak Scaling Laws for Large Language Models: Polynomial–Exponential Crossover

Abstract

Adversarial attacks can reliably steer safety-aligned large language models toward unsafe behavior. Empirically, we find that adversarial prompt-injection attacks can amplify attack success rate from the slow polynomial growth observed without injection to exponential growth with the number of inference-time samples. We first identify a minimal statistical mechanism for these two regimes by giving a small set of assumptions on the distribution of safe generation across contexts under which both scaling laws follow. To explain this phenomenon further, we propose a theoretical generative model of proxy language in terms of a spin-glass system operating in a replica-symmetry-breaking regime, where generations are drawn from the associated Gibbs measure and a subset of low-energy, size-biased clusters is designated unsafe. We analytically show how this model naturally realizes the minimal assumptions. Short injected prompts correspond to a weak magnetic field aligned towards unsafe cluster centers and yield a power-law scaling of attack success rate with the number of inference-time samples, while long injected prompts, i.e., strong magnetic field, yield exponential scaling. We observe qualitatively consistent behavior across a broad range of large language models, spanning parameter scales from 3B to 70B. In particular, the main trends remain stable across multiple attack methods, such as GCG and AutoDAN, as well as across benchmark datasets such as AdvBench and HarmBench.

Primary area:
learning theory
Keywords:
Jailbreaking LLMs, Inference-time scaling laws, AI safety and alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XM7BOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how jailbreak attack success rates scale with the number of independently generated responses per prompt. Starting from the distribution of prompt-conditioned single-response safety probabilities, the authors derive an expression for attack success under repeated sampling and propose a scaling form combining polynomial and exponential factors. A spin-glass model is further introduced to provide a theoretical interpretation of the scaling behavior under different attack strengths. The empirical evaluation covers GCG, AutoDAN, and no-injection settings across multiple language models and two benchmarks, AdvBench and HarmBench, together with comparisons of evaluation methods and attack-category analyses. The results show that prompt injection increases attack success within the evaluated settings and that the proposed scaling form provides good fits to the observed curves.

Critical strengths

S1. Broad empirical evaluation with complementary judge validation

The paper provides a reasonably broad empirical evaluation. Figures 3–4 and Appendix H cover multiple language models, two attack methods (GCG and AutoDAN), two benchmarks (AdvBench and HarmBench), and category-level analyses, offering evidence across different settings that prompt injection alters attack-success curves under finite sampling budgets. The authors also address potential evaluation bias arising from refusal-string-based metrics. Specifically, Appendix G.2 compares refusal-based and semantic evaluations across different sampling budgets, while Appendix G.4 examines agreement between two LLM-based judges. These complementary evaluations strengthen the credibility of the empirical findings beyond what could be established by a single model, attack configuration, or evaluation metric.

S2. An informative probabilistic framework supported by theoretical-model simulations

The paper develops an informative probabilistic framework for analyzing jailbreak success under repeated sampling. By expressing the probability of no successful jailbreak after k independent generations as E[Pᵏ], the analysis connects the scaling behavior to heterogeneity across prompt-conditioned safety probabilities and the upper-tail properties of their distribution. This offers a richer perspective than evaluating single-response attack success rates alone. In addition, the numerical simulations in Appendix F, particularly the comparison with analytical predictions in the weak-field regime, provide supporting evidence for the behavior predicted within the theoretical model. Although the connection between the spin-glass model and actual LLM behavior remains primarily qualitative, the proposed framework offers a valuable theoretical perspective on repeated-sampling jailbreak dynamics.

Critical weaknesses

W1. Insufficient evidence for the necessity and robustness of the exponential term

A central empirical claim of the paper relies on the proposed polynomial–exponential scaling form, yet the current analysis does not sufficiently establish the necessity or robustness of the exponential component. Appendix G.5.3 fits the transformed quantity log(−log Πₖ) using ordinary least squares and reports high R² values. However, goodness of fit alone does not demonstrate that the model with an exponential component (μ > 0) provides a statistically or predictively meaningful improvement over a pure power-law model (μ = 0), nor does it establish that the observed finite-budget trends represent an asymptotic polynomial-to-exponential crossover. The paper does not report direct comparisons against the simpler model, uncertainty estimates for the key parameters μ and ν, or out-of-range predictive validation using larger sampling budgets. Although Appendix G.5.1 reports two independent generation sets and corresponding ASR statistics, these do not directly quantify uncertainty in the fitted scaling parameters.

Furthermore, the caption of Figure 3 acknowledges a numerical limitation around 1 − Πₖ ≈ 10⁻³ for high-injection curves. As ASR approaches one, the fitted transformation may become particularly sensitive to estimation errors, potentially affecting the reliability of tail behavior and parameter estimates. The manuscript does not clearly specify whether these boundary-affected points are included in the fits or whether the central conclusions remain stable under alternative treatments. While this does not imply that the exponential component is a numerical artifact, its sensitivity to these limitations remains insufficiently examined. Direct comparisons between the pure power-law and mixed models, together with parameter uncertainty estimates and boundary-sensitivity analyses, would substantially strengthen the evidence for the claimed crossover. (pp. 9, 65–66)

W2. Limited empirical support for the proposed distributional assumptions and mechanistic interpretations

The empirical evidence connecting the theoretical assumptions to actual LLM behavior remains limited. First, the proposed exponential decay is associated with an upper endpoint strictly below one in the distribution of prompt-conditioned single-response safety probabilities. Although the finite-sample histograms in Figure 1 are qualitatively consistent with this assumption, they do not establish the essential supremum of the underlying distribution or directly validate its near-endpoint tail behavior. Moreover, the observed finite-budget scaling curves may be influenced by the discrete nature of the evaluated prompt set and uncertainty in estimated single-response safety probabilities. Consequently, obtaining good fits to the proposed scaling form is distinct from empirically validating the distributional assumptions required by the theoretical analysis. (p. 2)

Second, Section 5 relates the fitted parameters to reasoning depth and adversarial order, while using injection length as a proxy for effective field strength. Although Figure 2 provides some supporting trends, the evaluation includes only three injection-length settings, does not report parameter uncertainty, and does not exhibit a consistently monotonic trend in ν across all models. More importantly, the paper does not independently measure reasoning depth or reasoning ability, making it difficult to determine whether changes in the fitted parameters genuinely correspond to changes in these properties. The results therefore support a qualitative correspondence between the theoretical framework and empirical observations, but fall short of establishing the proposed mechanistic relationships. The authors should more clearly distinguish theoretical interpretations from experimentally validated conclusions and qualify the corresponding mechanistic claims accordingly. (pp. 7–8)

Questions for authors

See Weeknesses

1 comment

We are very much thankful to the reviewer for the time and the precise set of questions. We address the concerns with new analyses below.

How the floor is chosen. We thank the reviewer for pressing on this. With M=520M=520 prompts in AdvBench, 1−Π^k1-\hat\Pi_k is quantized in steps of 1/M=1.9×10−31/M=1.9\times10^{-3}. This is why our code clips 1−Π^k1-\hat\Pi_k at 10−310^{-3} where finite size effects become important and log⁡(−log⁡(1−1/520))=−6.25\log(-\log(1-1/520))=-6.25 is what determines the plateau in Fig. 3, 4. We have added this discussion to Fig. 3 caption in the revised version of the paper.

Uncertainty on ν^,μ^\hat\nu,\hat\mu. In the paper, the goodness of the fit is measured by R2R^2, which is calculated only from points that are above the floor value. We agree with the reviewer that this does not tell us about uncertainty in the fitted parameters in repeated trials. In the plots, we have already reported variance from repeatedly sampling the same LLM at a given temperature. In addition, we now bootstrap the entire pipeline over prompts resampling (20002000 times) the 520 AdvBench prompts with replacement, recomputing Πk\Pi_k):

ModelAttackν^\hat\nu [95% CI]μ^\hat\mu [95% CI]p^=e−μ^\hat p=e^{-\hat\mu} [95% CI]
Llama-3-8B-Instructnone0.335 [0.302, 0.371]0.010 [0.008, 0.011]0.990 [0.989, 0.992]
Llama-3-8B-InstructGCG0.354 [0.253, 0.646]0.157 [0.093, 0.180]0.855 [0.835, 0.911]
Llama-3-8B-InstructAutoDAN0.898 [0.736, 1.111]0.053 [0.011, 0.093]0.948 [0.911, 0.989]
OLMo-2-0325-32B-Instructnone0.172 [0.138, 0.214]0.0012 [0.0003, 0.0020]0.9988 [0.9980, 0.9997]
OLMo-2-0325-32B-InstructGCG0.582 [0.440, 0.827]0.124 [0.072, 0.158]0.883 [0.854, 0.930]
OLMo-2-0325-32B-InstructAutoDAN0.633 [0.584, 0.685]0.0113 [0.0083, 0.0151]0.989 [0.985, 0.992]

We have now added a detailed section on statistical measures of the fitted parameters in Appendix I.2 (confidence interval measure) and Appendix I.3 (nested comparison, μ=0\mu=0 vs μ>0\mu>0, see our reply to reviewer 9BKQ) in the revised paper and refer them in section 5 of main text.

What a finite-sized experiment can and cannot establish. We totally agree that no finite-kk experiment can establish an asymptotic crossover, and we now state the limitation quantitatively. Let FF be any law of the per-prompt safe probability PP with p^=e−μ^<1\hat p=e^{-\hat\mu}<1 satisfying Assumption 1, and write gF(k)=EF[Pk]=1−Πkg_F(k)=\mathbb E_F[P^k]=1-\Pi_k. Add a fraction ww of prompts with p^=1\hat p=1: Gw=(1−w)F+wQνG_w=(1-w)F+wQ_\nu with P(Qν≥1−ε)=εν\mathbb P(Q_\nu\ge1-\varepsilon)=\varepsilon^\nu. Then: let k∗(w):=min⁡(k≥1: w gQ(k)≥gF(k))k^\ast(w):=\min(k\ge1:\ w\,g_Q(k)\ge g_F(k)), with gQ(k)=E[Qνk]g_Q(k)=\mathbb E[Q_\nu^k], be the first kk at which the added component overtakes the original curve. Because gQ(k)≤1g_Q(k)\le1, this can happen only after the original curve has fallen below ww: k∗(w)≥min⁡{k:gF(k)≤w}k^\ast(w)\ge\min\{k:g_F(k)\le w\}. gF(k)g_F(k) sees a finite size effect when it is of order 1/M1/M, and hence w<1/Mw<1/M is not ruled out by our experiments. We now state the limitation quantitatively in Appendix I.1 of the revised paper.

μ\mu, ν\nu as a function of attack strength. The theory predicts monotone growth of ν\nu only in the weak-field regime (Remark 2) - not in general. The corresponding LLM Fig. 2 also shows the growth of ν^\hat\nu in the small jailbreak strength regime - so the later non-monotonicity is consistent in the theoretical model. In the strong-field regime, the leading-order theory fixes the exponential rate and as the magnetic field increases, that increases as shown in Appendix B, and says nothing about the pre-factor exponent. In the corresponding LLM Fig. 2, we see that at large jailbreak strength μ^\hat\mu does increase significantly. However, we agree with the reviewer on the criticism that we have not performed an extensive amount of experiments focused on understanding these - in particular, one can perform the same experiment at different points of GCG optimization strings and use it as the proxy of variable jailbreak strength. Also, we agree that the sentences interpreting ν^\hat\nu as "lack of reasoning" following the Souly et al. capability observation are speculative, and we have moved this speculation to the caption of Fig. F.1.

SpinLLM vs realistic LLM. Please see our response to see our reply to reviewer j2x2.

Thank you so much for the review, please let us know if there are more questions.

Reviewer_9BKQOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

For the record: I did not finished the appendix since it is a 70+ pages paper.

The paper studies how jailbreak attack success changes with the number of sampled LLM generations under prompt injection. It derives a general upper-edge-tail result yielding a power-law times exponential ASR-gap form, and proposes a spin-glass proxy model in which injection acts as a field toward unsafe clusters. Experiments on Llama and OLMo models with GCG and AutoDAN show qualitatively different scaling curves as injection increases.

Critical strengths

The minimal statistical characterization is crisp and useful. Section 2 and Result 2 identify that the relevant quantity is E[Pk]\mathbb{E}[P^k], where P is the context-dependent safe-generation probability, and show that an upper-edge law yields 1−Πk∼c^Γ(v^+1)e−kμ^k−v^1-\Pi_k \sim \hat{c}Γ (\hat{v} +1) e^{-k\hat{\mu}}k^{-\hat{v}}.This cleanly separates a general probabilistic statement from any particular mechanistic model, and is the paper’s most durable contribution.

The empirical phenomenon is practically relevant and appears across two attack constructions. Figures 3 and 4 show qualitatively similar curvature changes under both GCG and AutoDAN for Llama-3-8B-Instruct and OLMo-2-0325-32B-Instruct. The use of an LLM judge rather than refusal-string matching, motivated in Section 5, is also important because the paper documents that refusal-string metrics can count incoherent or irrelevant outputs as successes.

Critical weaknesses

The strong-field spin-glass analysis does not establish the advertised exponential-in-k ASR law. In Section 4.2, Result 5 gives only a lower bound,

1kNlog⁡(1−Πk)≥−Ih(ql+1).\frac{1}{kN}\log(1-\Pi_{k})\geq-I_{h}(q_{l+1}).

The subsequent claimed matching upper bound relies on treating Equation (22) as a "weak version" of Assumption 10 in Remark 3 떼, with p^=e−N(I⋆−δ)\hat{p}=e^{-N(I_{\star}-\delta)} and ν^=0.\hat{\nu}=0. But Equation (22) only bounds the probability that PN(h,J)P_N(h,J) exceeds a threshold near e−N(L+−δ);e^{-N(L_+-\delta)}; it does not establish that this threshold is the essential supremum of PNP_N, nor that PN/p^∈[0,1].P_N/\hat{p}\in[0,1]. Those properties are required in the proof of Result 9, which explicitly sets Q=P/p^∈[0,1]Q=P/\hat{p}\in[0,1] and uses the resulting tail-integral decomposition. Thus the stated concentration inequality cannot directly supply the upper bound needed to combine with Result 5 and conclude

log⁡(1−Πk)/(kN)=−Ih(ql+1).\log(1-\Pi_k)/(kN)=-I_h(q_{l+1}).

This matters because the weak-field/strong-field polynomial-to-exponential crossover is the central theoretica claim of SpinLLM, not a peripheral corollary. A resolution would require either proving an appropriate deterministic endpoint bound and matching edge condition for PNP_N, or weakening the strong-field conclusion to the one-sided result actually established.

The LLM experiments demonstrate fitted curves, not a validated crossover mechanism or field interpretation Section 5 u fits each condition to log⁡(−log⁡Πk)=−ν^log⁡k−μ^k+log⁡C^\log(-\log\Pi_k)=-\hat{\nu}\log k-\hat{\mu}k+\log\hat{C}, then interprets increasing fitted ν^\hat{\nu} and μ^\hat{\mu} with injection length as support for the magnetic-field model. Figures 3 0 and 4, however, provide neither confidence intervals nor a held-out model comparison between a pure power law and a power- times- exponential law, while the high-injection curves are explicitly censored by a numerical floor at 1−Πk≈10−3.1-\Pi_k\approx10^{-3}. With three flexible fit parameters per curve, qualitative curvature alone does not establish an exponential component or dentify injection length with a field strength. This is consequential because the paper's main empirical-to- theoretical bridge rests on this interpretation. The authors could substantially address this with existing data by reporting uncertainty over prompts/seeds, fitting on a subset of kk values and evaluating predictive likelihood on held-out kk, and comparing the nested μ^=0\hat{\mu}=0 model against μ^>0\hat{\mu}>0 under a stated selection procedure

Questions for authors

For Figures 2 to 4, what is the estimation procedure for (ν^,μ^,C^)(\hat{\nu},\hat{\mu},\hat{C}), including the number of prompts, samples, seeds, fitting range in kk, and uncertainty estimates? In particular, does a held-out comparison favor the μ^>0\hat{\mu}>0 model over the nested μ^=0\hat{\mu}=0 model before the 10−310^{-3} numerical floor is reached? Evidence of this kind would materially strengthen the empirical crossover claim.

1 comment

We are very much thankful to the reviewer for reading the paper with great care and the constructive comments. Please see our response below.

Strong-field upper bound.

We are grateful to the reviewer for pointing out the mathematical gap in our argument. We have found an alternate proof as explained below.

Result: under EYN→I⋆\mathbb EY_N\to I_\star and Assumption 6 (sub-Gaussian self-averaging with constant c0c_0), for every fixed kk, −I⋆≤lim⁡Nlog⁡(1−Πk)kN≤−I⋆+k4c0.-I_\star\le\lim_{N}\frac{\log(1-\Pi_k)}{kN}\le-I_\star+\frac{k}{4c_0}.

For a fixed prompt JJ, the kk samples are independent and each is safe with probability PNP_N, so all kk are safe with probability PNkP_N^k. Averaging over prompts, and using PNk=(e−NYN)k=e−kNYNP_N^k=(e^{-NY_N})^k=e^{-kNY_N}, 1−Πk=E[PNk]=E[e−kNYN].1-\Pi_k=\mathbb E\big[P_N^k\big]=\mathbb E\big[e^{-kNY_N}\big]. The function φ(y)=e−kNy\varphi(y)=e^{-kNy} is convex, hence using Jensen's inequality: 1−Πk=E[e−kNYN] ≥ e−kNEYN.1-\Pi_k=\mathbb E\big[e^{-kNY_N}\big]\ \ge\ e^{-kN \mathbb EY_N}. The logarithm is increasing, so taking logarithms keeps the inequality; dividing by kN>0kN>0 and letting N→∞N\to\infty gives the left inequality.

Write YN=EYN−XY_N=\mathbb EY_N-X. Then e−kNYN=e−kNEYNekNXe^{-kNY_N}=e^{-kN \mathbb EY_N} e^{kNX}. The first factor is a number, not a random variable, so it comes out of the average: 1−Πk=e−kNEYN E[ekNX].1-\Pi_k=e^{-kN \mathbb EY_N}\ \mathbb E\big[e^{kNX}\big]. It remains to show that E[esX]\mathbb E[e^{sX}] at s=kNs=kN is not too large. This is the moment generating function of XX. For any non-negative random variable WW, E[W]=∫0∞P(W>u) du\mathbb E[W]=\int_0^\infty\mathbb P(W>u)\,du. Apply this to W=esXW=e^{sX} with s>0s>0, and split the integral at u=1u=1:

  • for 0<u≤10<u\le1, bound P(esX>u)≤1\mathbb P(e^{sX}>u)\le1; this part contributes at most 11;
  • for u>1u>1, substitute u=estu=e^{st} with t>0t>0, so du=sestdtdu=s e^{st} dt; since the exponential is increasing, {esX>est}={X>t}\{e^{sX}>e^{st}\}=\{X>t\}. Hence E[esX] ≤ 1+∫0∞sestP(X>t)dt.\mathbb E\big[e^{sX}\big]\ \le\ 1+\int_0^\infty s e^{st} \mathbb P(X>t) dt . Performing the integral, E[ekNX] ≤ 1+2kπNc0ek2N/(4c0) ≤ (1+2kπNc0)ek2N/(4c0).\mathbb E\big[e^{kNX}\big]\ \le\ 1+2k\sqrt{\frac{\pi N}{c_0}} e^{k^2N/(4c_0)}\ \le\ \Big(1+2k\sqrt{\frac{\pi N}{c_0}}\Big) e^{k^2N/(4c_0)} . Combining and taking logarithms: log⁡(1−Πk)kN ≤ −EYN+k4c0+1kNlog⁡(1+2kπNc0).\frac{\log(1-\Pi_k)}{kN}\ \le\ -\mathbb EY_N+\frac{k}{4c_0}+\frac{1}{kN}\log\Big(1+2k\sqrt{\frac{\pi N}{c_0}}\Big). As N→∞N\to\infty: EYN→I⋆\mathbb EY_N\to I_\star, the middle term does not depend on NN; the last term is log⁡(const⋅N)/(kN)→0\log(\text{const}\cdot\sqrt N)/(kN)\to0. This gives the right inequality.

On broad general grounds we assume I⋆I_\star to grow with hh - in Appendix B we check this explicitly with a linear growth. For k≪4c0I⋆k\ll4c_0I_\star, we have log⁡(1−Πk)=−kNI⋆\log(1-\Pi_k)=-kNI_\star. We added this in the revised version of the paper replacing Result 7 and Remark 3.

Estimation procedure, uncertainty, held-out model comparison

Comparison of μ=0\mu=0 vs μ>0\mu>0.

Guided by the reviewer's suggestion we performed this test: at each kk we count the number of prompts whose kk samples were all judged safe, SkS_k. Each kk is a separate generation run, so given the prompt set the counts at different kk are independent, and theoretically we expect Sk∼Bin(M,1−Πk)S_k\sim\mathrm{Bin}(M,1-\Pi_k) with 1−Πk=1−exp⁡{−exp⁡(log⁡C−νlog⁡k−μk)}1-\Pi_k=1-\exp\{-\exp(\log C-\nu\log k-\mu k)\}. We consider H0: μ=0\mu=0 and H1: μ≠0\mu\neq0 and use binomial maximum likelihood ℓ = Σₖ [Sₖ·log(1 − Πₖ) + (M − Sₖ)·log Πₖ] on the kk-values with Sk≥5S_k\ge5 to evaluate them (these lie before the resolution floor). We compare the two fits by the one-sided likelihood-ratio statistic Λ=2(ℓ1−ℓ0)\Lambda=2(\ell_1-\ell_0), with a parametric-bootstrap pp-value (= ½·P(χ²₁ > Λ)), and by ΔAIC = Λ − 2 and ΔBIC = Λ − ln n, with n = M × (number of kept k-values).

ModelAttackkk used (points)ℓ0\ell_0ℓ1\ell_1Λ\LambdappΔAICΔBIC
OLMo-2-0325-32B-Instructnone1–125 (32)−6960.6−6958.73.80.0301.8−5.9
OLMo-2-0325-32B-InstructGCG1–25 (7)−1135.8−1111.349.0< 0.00147.040.8
OLMo-2-0325-32B-InstructAutoDAN1–125 (47)−9870.5−9808.4124.2< 0.001122.2114.1

We have now added a detailed section on statistical measures of the fitted parameters in Appendix I.2 (confidence interval measure, see our reply to reviewer XM7B) and Appendix I.3 (nested comparison, μ=0\mu=0 vs μ>0\mu>0) in the revised paper and refer them in section 5 of main text.

Sampling Procedure. M=520M=520 AdvBench prompts (200 for HarmBench); k∈{1,…,20}∪{21,25,…,125}k\in\{1,\dots,20\}\cup\{21,25,\dots,125\}, sampling temperature 0.6, no top-p or top-k is set, 256 new tokens, seed 1234, judge Mistral-7B-Instruct-v0.3 with greedy decoding, score ≤4\le4 = jailbroken, kk generations were drawn afresh per kk, two independent generation sets. We have explicitly clarified all these details in Appendix G.5.1 in the revised paper.

Floor, finite size effects. Appears in Fig. 3 caption and Appendix I.1 in revised version of the paper. Also, see our reply to reviewer XM7B.

We thank the reviewer again for recommending acceptance.

Reviewer_j2x2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper is about how fast an attacker can manage to get an unsafe answer from a large language model as it tries to repeat the same attack many, many times. It's trying to quantify this phenomenon under both settings, with and without adversarial suffixes, such as those you would get with GCG or Toden. There are two main theoretical contributions. The first one is a new model to explain this phenomenon. Given a number of samples k, they characterize the fraction of prompts where all the k answers remain safe. Their model shows that the shape of that fraction, which is an average over prompts, will depend on how close it is to the safest prompts possible. When the prompt is really strong and almost never fails, then their model shows that it's going to look like a par law in K. But interestingly, they say that if it happens that even the safest prompt has somehow a chance of failing, then a better model would add an exponential term. The second theoretical contribution is an attempt to use the spinless model to better characterize that phenomenon beyond the power-law exponential model that they propose. They view potential responses as forming "unsafe" clusters under the influence of a magnetic field that is going to tilt or bias the landscape. They show that if that magnetic field is weak, then in that case it gives rise to a power law, and when the field is stronger, it goes towards an exponential. They fit their parallel times exponential model to empirical data coming from Llama and Olmo models ranging from 3 to 7 billion parameters.

Critical strengths

While I'm not an expert in that field, I think that the spinglass model is interesting and promising and might provide a useful vocabulary to capture an important phenomenon. The question is definitely relevant, and repeated sampling can be seen as pretty cheap in a realistic attack type that an attacker could deploy easily. The evaluation looks well-taught. They've used LLM judges rather than using explicit refusal strings, and they verify afterwards whether two judges agree.

Critical weaknesses

Now, I think that power laws tend to be quite ubiquitous, and I think we need to be careful in claiming that power laws are the thing we care about. I also invite the authors to have a look at Cosma Shalizi's great writings on that topic. And in that sense, I think there is a significant overlap with one of the papers that you cite in the related work, in particular the paper by Schaeffer and colleagues (2025): "How do large language monkeys get their power laws?" Correct me if I'm wrong, but I believe that the model you propose might be a slight generalization of theirs. Schaeffer and colleagues studied the question of repeated sampling in the general sense, not necessarily in the adversarial setting. I think that the same mechanism is pretty much present there and shows the emergence of power laws as you repeat.

Well, I think that the spin glass model it's quite original, it's not developed to the extent that we would expect. At least the thing I would have liked to see is a setting where you try to fit it to data, the same way that you've been able to fit the power-law-times-exponential model to experimental data. It would have been really nice to see the same thing happening with the spin glass model. My understanding of what you do with that model is mostly to provide insight and intuition, but it's not really an actionable model that you can fit to the data or derive new theoretical insight from. More as an inspiration than anything. So it'd be nice if you could actually tie actual quantities inside a real network with the knobs on your model, such as the field strength, temperature, or the depth of the cluster hierarchy. There might also be a bit of a discrepancy with what Figure 2B is saying versus what the model offers. It seems to work in one direction: as you get a longer suffix, the field gets stronger, and fitted numbers move that way. But it seems that the power-law exponent could also go to zero under the framework of strong field theory, while in Figure 2b it seems to rise with the suffix length rather than going to zero.

Questions for authors

I think it would be important that you describe a little bit better the connection to Schaeffer. It's an important related work, and although you cite it, I think you need to expand beyond that.

I also have a question regarding the experiments. There are really few points that are being used. I understand that these are costly experiments. That being said, to make proper experimental claims, we might need a bit more than that, and actually, it'd be good to see confidence intervals.

In Figure 1A, which one is exactly GPT Turbo 4.5, and what were the settings used for sampling?

1 comment

We are very much thankful to the reviewer for reading the paper with great care and pointing out Shalizi's point of view. Please see our response below.

Relation to Schaeffer et al. (2025)

We totally agree with the reviewer that our work is an extension of Schaeffer et al. (2025). We have now prominently mentioned it in the introduction of the revised version of the paper and added Appendix J with a detailed comparison. With P=1−pass@1P=1-\mathrm{pass}@1, their Theorem 3.1 density ∝(pass@1)b−1\propto(\mathrm{pass}@1)^{b-1} near 0 ⇒ −log⁡pass@k∼CΓ(b)k−b-\log\mathrm{pass}@k\sim C\Gamma(b)k^{-b} is exactly our Result 2 at p^=1\hat p=1 with ν^=b\hat\nu=b and c^=C/b\hat c=C/b. Our Result 2 is therefore restated as "an extension of Theorem 3.1 of Schaeffer et al. to an upper endpoint p^<1\hat p<1", with Assumption 1 credited accordingly. We show that prompt injection based jailbreak can lead to a situation where p^<1\hat p<1 in Fig. 1 (c). In addition, Schaeffer et al. fit the per-problem single-attempt success probability with a Beta distribution. The natural extension to p^<1\hat p<1 would be Pi=cUiP_i=cU_i with Ui∼Beta(a,b)U_i\sim\mathrm{Beta}(a,b). The law gives the curve exactly: 1−Πk=E[Pk]=ckB(a+k,b)B(a,b).1-\Pi_k=\mathbb E[P^k]=c^k \frac{B(a+k,b)}{B(a,b)} . At large kk, once almost all prompts are broken, −log⁡Πk≃1−Πk-\log\Pi_k\simeq1-\Pi_k, so log⁡(−log⁡Πk)≃log⁡Γ(a+b)Γ(a)−blog⁡k−(−log⁡c)k,i.e.ν^→b,μ^→−log⁡c,C^→Γ(a+b)Γ(a).\log(-\log\Pi_k)\simeq\log\frac{\Gamma(a+b)}{\Gamma(a)}-b\log k-(-\log c) k,\qquad\text{i.e.}\qquad \hat\nu\to b,\quad \hat\mu\to-\log c,\quad \hat C\to\frac{\Gamma(a+b)}{\Gamma(a)} . We could not paste the detailed derivation/results here because of character limitations. Please see the details of it and experimental fits in Appendix J of the revised version of the paper.

Shalizi: "Power laws are ubiquitous", uncertainty, comparison of μ=0\mu=0 vs μ>0\mu>0, finite size effects

We are grateful to the reviewer for pointing out Shalizi's discussion and we agree with it. We now present statistically grounded tests to justify it more. Please see our reply to reviewer XM7B and Appendix I.2 of the revised paper for a discussion of uncertainty/CI of estimated parameters which show prompt injection increases exponential fall off over pure power law fit. We have also calculated ΔAIC and ΔBIC for comparison of μ=0\mu=0 vs μ>0\mu>0 fits, please see our reply to reviewer 9BKQ and Appendix I.3 of the revised paper. We have also presented limitation of our experiments due to finite size effects (Appendix I.1) to the response to reviewer XM7B. We agree that with limited costly experiments we cannot decisively establish power law in rigorous terms.

Confidence intervals. Please see our response to see our reply to reviewer XM7B and 9BKQ.

Sampling settings. Please see our reply to reviewer 9BKQ for the details of the sampling method. With the same method, we ran a full kk-sweep on Claude-Haiku-4.5 with no injection and with AutoDAN-Turbo, up to k=1000k=1000. This new result has been added as Fig. H.5 in the revised version of the paper.

SpinLLM vs realistic LLM. Here we present a motivational discussion connecting SpinLLM to realistic LLM as asked by the reviewer. Say we coarse-grain a NN token generation of a LLM into σ∈(−1,+1)N\sigma\in(-1,+1)^N, with σi=+1\sigma_i=+1 if segment ii is part of is safe and −1-1 if unsafe, as labelled by the judge. For prompts from a narrow text sub-domain, the LLM induces a probability distribution P(σ)P(\sigma) on (−1,+1)N(-1,+1)^N. The following two facts are exact:

  1. Every strictly positive law on (−1,+1)N(-1,+1)^N is a Gibbs measure, P(σ)=e−H(σ)/ZP(\sigma)=e^{-H(\sigma)}/Z with H=−log⁡P+constH=-\log P+\text{const}.
  2. HH has a unique Walsh–Fourier (multilinear) expansion, H(σ)=const−∑ihiσi−∑i<jJijσiσj−∑i<j<lKijlσiσjσl−⋯H(\sigma)=\text{const}-\sum_i h_i\sigma_i-\sum_{i<j}J_{ij}\sigma_i\sigma_j-\sum_{i<j<l}K_{ijl}\sigma_i\sigma_j\sigma_l-\cdots

At infinite sampling temperature of the LLM generation is uniform (i.e, only the constant term above is important), and hence near that, the higher-order coefficients are bound to be small. Keeping the lowest orders gives a 2-spin glass type model. Equivalently, this is the maximum-entropy model that matches the first and second moments of σ\sigma. The couplings depend on the sub-domain, i.e., on the context. Varying the context over the language distribution induces a distribution of (h,J)(h,J), which is the disorder of SpinLLM. Prompt injection pushes generations toward unsafe content along a low-dimensional direction (Arditi et al. 2024). At lowest order, it enters as a shift of the linear coefficients, hi→hi+h τih_i\to h_i+h\,\tau_i, which is the field term of eq. (15). A detailed investigation of this is left for future studies. We could add this motivational discussion in a revised version of the paper.

We thank the reviewer again for recommending acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.