- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 1
- Presentation
- 1(−1)
- Soundness
- 2
Read review
Paper summary
This paper studies the mechanism of BoN jailbreaking attacks against large language models, challenging existing accounts of power-law scaling and exponential crossover. It proposes a trap model inspired by statistical physics, where jailbreaking requires crossing an entropic bottleneck and a thermally activated energetic saddle in residual stream space. With four interpretable parameters (), the model predicts and collapses the two-budget attack surface. independent augmented prompts, each sampled for completions. It also captures the effect of decoding temperature .
Critical strengths
- The research question is meaningful, and the two-budget perspective is valuable.
- Beyond fitting existing attack surfaces, the authors test extrapolation with independent large-budget samples and predictions at unseen temperatures. These experiments provide initial evidence of the model’s practical predictive value.
Critical weaknesses
-
The four-parameter claim lacks a complete mathematical basis. Equation (3) specifies the marginal distributions of and , but not their dependence, so it cannot uniquely determine the attack surface. Under independence, Equation (9) requires . Yet Table 1 reports for Qwen-3B. This inconsistency needs clarification, otherwise both parameter sufficiency and the basis for extrapolation remain unclear.
-
The empirical fit does not directly validate the proposed physical mechanism. The theory uses a hard boundary and probability clipping, while the fitting model uses a Beta distribution with a soft wall and recovers physical parameters through mean matching. Matching means does not ensure agreement in higher-order moments, which directly determine the attack surface for . Several later derivations also ignore clipping. Thus, a good Beta fit does not establish the validity of the exponential barrier model.
-
The experiments do not isolate finite dataset size as the sole cause of the exponential crossover. Figure 7(c) enlarges the dataset by adding harder prompts, changing both sample size and difficulty distribution. The crossover shift therefore cannot be attributed to sample size alone. Although a finite mixture eventually produces an exponential tail, this does not exclude alternatives such as a positive lower bound on the overall success rate. The evidence is insufficient to refute existing explanations.
-
Filtering hard prompts undermines the tail-scaling and safety comparisons. The authors remove prompts with zero or few successes, as well as those with fitting anomalies. Only 80 of 100 Llama prompts remain. Zero-success prompts still constrain success probabilities and should not be treated as uninformative. Since large-budget behavior depends heavily on the hardest prompts, this filtering distorts the critical tail distribution. Bootstrapping the retained samples cannot correct the selection bias, and the paper does not show that its main conclusions survive proper treatment of the excluded prompts.
Questions for authors
please check the Weaknesses
3 comments
The authors fit a Beta distribution and show it predicts success rates when they sample more answers per prompt variation. What remains open is whether the proposed exponential barriers explain those rates. The exponential tail adds little evidence for that mechanism. A finite dataset can produce such a tail simply because the hardest prompt eventually accounts for most of the remaining failures. So accurate labels on the hardest prompts matter a lot. For the 20 hardest prompts, human reviewers checked 97 answers flagged by Granite alone and judged only two to be harmful. Judge errors could therefore shift the point where the curve turns exponential. I see value in studying both sampling budgets at different temperatures. Still, I find the paper's novelty and evidence limited and will keep my original score.
Thank you for the follow-up. We respectfully disagree with this assessment, and answer the points below.
Exponential barriers. The paper does not rest on the exponential form, and the revised text does not claim that the Beta fit establishes it. The mechanism we propose is that each augmentation selects a barrier which is then crossed thermally, on top of an entropic bottleneck set by the prompt. Three tests support it beyond the quality of the Beta fit. (i) The barrier can be read from the residual of the augmented prompt alone, before any completion is generated (held-out up to 0.76, App. H). (ii) Moving an augmentation so that this barrier is lowered or raised moves the jailbreak rate from 5.6% to 16.4% or to 2.2%, with a slope of 0.93 nats per standard deviation against 1.03 for the correlation, while the same move in a random direction does nothing (App. I). (iii) The ASR at four unseen temperatures is predicted with an error 3.7 to 7.5 times smaller than pure Arrhenius (Fig. 5). Moreover, the results in do not depend on the law of the barriers at all. At the law of within a prompt enters only through its mean, so the extrapolation in , the drifting exponent and the crossover rely only on the Gaussian law of the effective barrier across prompts.
Exponential tail. We never use the exponential tail as evidence for the mechanism. Sec. 4.3 shows that it is what a finite dataset produces once its hardest prompt carries the remaining failures, which is the statement you make. We also test it: on random subsets of prompts the measured resolution budget is 52, 65, 75, 85, 111 and 111 for to , and the fitted law predicts 49, 62, 72, 84, 96 and 108 with no additional parameter (Fig. 7c).
Labels of the hardest prompts. The judge sets where the crossover happens, and the paper says so (Sec. 4.3 and App. G). A judge which triggers more easily simply runs into the finite-size effect first, but any judge will eventually run into it. So yes, "judge errors" do shift the point where the curve turns exponential. What we provide is a way to measure and predict where this crossover happens for any judge (even a perfect human annotator), namely at . With HarmBench labels the law predicts on Qwen-3B, and indeed there is no crossover over the whole arm (Fig. 7(a-b)).
Novelty and evidence. To our knowledge this is the first model of the full attack surface and of its temperature dependence. Five numbers fitted at predict the survival at within 0.02–0.09 decades under three label sets, against 0.10–0.29 for the best existing baseline, and they remain predictive on held-out prompts and on a second dataset (App. J). To our knowledge it is also the only model which predicts the ASR at other temperatures zero-shot. Each of its five constants has a physical interpretation, which gives a mental picture of how the defense of an LLM is organized.
Thank you for a careful and technical reading of our work. Your first point in particular identified a flawed assumption in how we specified the model, and your 4th point led us to use a better estimator to obtain our modelling parameters. Fixing these two concerns has reduced the extrapolation error of our model by a factor of 3 to 6 and hence considerably improved our paper.
You are completely right. Eq. (3) of the submission only specified the marginals, and our population curves drew αb and logβb independently. The correlation looked negligible in our per-prompt estimates for the other models, and we wrongly assumed the same for Qwen-3B. The new fit over all prompts (see point 4) gives ρ=+0.48±0.04 there (2ΔlogL=87), and shows that the other models are correlated too, with ρ between −0.22 and −0.52 (significant for all but Gemma). Since even its sign changes across models, ρ is now a free parameter of the model.
A positive ρ lowers σE below σα (Eq. 10 of the revised manuscript), which is exactly the inconsistency you spotted.
In the revised manuscript the ansatz is now a bivariate Gaussian for (αb,logβb) (Eq. 3), and Table 1 has a ρ column. All population curves are recomputed with ρ. The error in log10 survival on Qwen-3B goes from 0.58 in the submission to 0.12 with ρ, and to 0.09 with ρ and the new estimator (point 4). On Qwen-7B-Instruct it goes from 0.26 to 0.21 and 0.09.
We tested the effect of clipping and saw negligible differences. Using the exact clipped mean, ⟨p⟩a=1−eβbTαb/(1+βbT) for αb<0, changes ASR@N by at most 0.012 for Llama and by at most 0.003 for every other model and every N (App. C).
On the higher-order moments: mean matching is only used to define αb (Eq. 21). The attack surface at M>1 is computed from the Beta law itself (Eq. 22 and App. B), so its higher moments are those of the fitted Beta and not of the hard-edged law. They are tested by the extrapolation in M, from M≤40 to M∼300 (Fig. 4b). The Beta is the within-prompt law we actually use, and the hard wall is its motivation.
We agree that the Beta-fit does not directly validate the exponential form. It simply supports that the exponential form is consistent with the data. Some other distribution might be a more accurate depiction of the actual behavior of the model. We rephrased our claims accordingly. We used an exponential to be able to make analytical predictions for the ASR and because it had the clearest physical interpretation. However, the core message that defense is organized through entropic (independent of T) and enthalpic (proportional to 1/T) channels does not rely on the choice of distributions for the saddles and/or αb,logβb. It is tested directly by the temperature experiment (Fig. 5). The modelling choices we made should be seen as a first, most obvious, guess. Not a definitive truth.
The revised Fig. 7(c) uses 300 random P-subsets of the 400 prompts for each P, so that only P changes and we don't influence the difficulty directly. Each subset follows the population law until its own hardest prompt takes over, and this break-away moves to larger N as P grows. Quantitatively, on Qwen-3B (Granite labels) the median resolution budget Nres=1/minbqb is 52, 65, 75, 85, 111 and 111 for P=10,25,50,100,200,400, while the extreme-value prediction of the fitted law, which has no floor, gives 49, 62, 72, 84, 96 and 108. A floor qmin>0 would instead pin Nres near 1/qmin. This excludes a floor above qmin≈0.009. A lower one cannot be excluded with 400 prompts, which is why we now write that the crossover is "consistent with" a finite-size effect. Any judge will eventually run into the finite-size effect, and a judge which triggers more easily simply runs into it first. With the stricter HarmBench labels the law holds over the whole 104 arm with no crossover (Fig. 7(a-b)), as expected since it predicts Nres≈2×104. We have also replaced "fake crossover" by "finite-size crossover" throughout.
We agree and have revised our fitting mechanism to address this issue (general response point 2). The five parameters are now fitted by maximizing the marginal likelihood over all prompts, with the per-prompt (αb,βb) integrated out (Eq. 22), so that never-jailbroken prompts simply enter as censored observations. The submitted per-prompt moments are kept for comparison in App. C (Table 2). This improved estimator led to lower errors on predicted ASR for all models. The correction is largest for Llama, whose μE rises from 2.05 to 2.75, and the comparison between models of Sec. 4 is unchanged.