acceptodds
Under review as a conference paper at ICLR 2027

Escaping Alignment: A Physical Trap Model of Best-of-N Jailbreaking

Abstract

Best-of- jailbreaking (BoN) bypasses safeguards of aligned models by drawing independent augmentations of an unsafe prompt and sampling completions of each. Previous works have shown that the attack success rate (ASR) seems to follow a power-law in , which we challenge. The exponent drifts with , with an exponential crossover which is a finite-size artifact of the adversarial dataset. Little work has been done to explore the entire two-budget () attack surface as well as its dependence on the generation temperature . We introduce a simple barrier model where each prompt has a baseline safety level and each augmentation a random thermally activated barrier. Then four numbers, each backed by an interpretable safety mechanism, determine the entire () attack surface. They extrapolate predictions from to , collapse five distinct models on the same scaling function and predict ASR at different temperatures from the one they were fitted at.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Best-of-N jailbreaking, LLM Safety, scaling laws, interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4vZ5OpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
1
Presentation
1(−1)
Soundness
2
Read review
Paper summary

This paper studies the mechanism of BoN jailbreaking attacks against large language models, challenging existing accounts of power-law scaling and exponential crossover. It proposes a trap model inspired by statistical physics, where jailbreaking requires crossing an entropic bottleneck αb\alpha_b and a thermally activated energetic saddle ηb,a\eta_{b,a} in residual stream space. With four interpretable parameters (μα,σα,μβ,σβ\mu_\alpha, \sigma_\alpha, \mu_\beta, \sigma_\beta), the model predicts and collapses the two-budget attack surface. NN independent augmented prompts, each sampled for MM completions. It also captures the effect of decoding temperature TT.

Critical strengths
  1. The research question is meaningful, and the two-budget perspective is valuable.
  2. Beyond fitting existing attack surfaces, the authors test extrapolation with independent large-budget samples and predictions at unseen temperatures. These experiments provide initial evidence of the model’s practical predictive value.
Critical weaknesses
  1. The four-parameter claim lacks a complete mathematical basis. Equation (3) specifies the marginal distributions of αb\alpha_b and log⁡βb\log\beta_b, but not their dependence, so it cannot uniquely determine the attack surface. Under independence, Equation (9) requires Var⁡(Eb)≥σα2\operatorname{Var}(E_b) \ge \sigma_\alpha^2. Yet Table 1 reports σE=0.58<σα=0.65\sigma_E = 0.58 < \sigma_\alpha = 0.65 for Qwen-3B. This inconsistency needs clarification, otherwise both parameter sufficiency and the basis for extrapolation remain unclear.

  2. The empirical fit does not directly validate the proposed physical mechanism. The theory uses a hard boundary and probability clipping, while the fitting model uses a Beta distribution with a soft wall and recovers physical parameters through mean matching. Matching means does not ensure agreement in higher-order moments, which directly determine the attack surface for M>1M > 1. Several later derivations also ignore clipping. Thus, a good Beta fit does not establish the validity of the exponential barrier model.

  3. The experiments do not isolate finite dataset size as the sole cause of the exponential crossover. Figure 7(c) enlarges the dataset by adding harder prompts, changing both sample size and difficulty distribution. The crossover shift therefore cannot be attributed to sample size alone. Although a finite mixture eventually produces an exponential tail, this does not exclude alternatives such as a positive lower bound on the overall success rate. The evidence is insufficient to refute existing explanations.

  4. Filtering hard prompts undermines the tail-scaling and safety comparisons. The authors remove prompts with zero or few successes, as well as those with fitting anomalies. Only 80 of 100 Llama prompts remain. Zero-success prompts still constrain success probabilities and should not be treated as uninformative. Since large-budget behavior depends heavily on the hardest prompts, this filtering distorts the critical tail distribution. Bootstrapping the retained samples cannot correct the selection bias, and the paper does not show that its main conclusions survive proper treatment of the excluded prompts.

Questions for authors

please check the Weaknesses

3 comments

Thank you for a careful and technical reading of our work. Your first point in particular identified a flawed assumption in how we specified the model, and your 4th point led us to use a better estimator to obtain our modelling parameters. Fixing these two concerns has reduced the extrapolation error of our model by a factor of 3 to 6 and hence considerably improved our paper.

  1. Modelling specification

You are completely right. Eq. (3) of the submission only specified the marginals, and our population curves drew αb\alpha_b and log⁡βb\log\beta_b independently. The correlation looked negligible in our per-prompt estimates for the other models, and we wrongly assumed the same for Qwen-3B. The new fit over all prompts (see point 4) gives ρ=+0.48±0.04\rho=+0.48\pm0.04 there (2Δlog⁡L=872\Delta\log L=87), and shows that the other models are correlated too, with ρ\rho between −0.22-0.22 and −0.52-0.52 (significant for all but Gemma). Since even its sign changes across models, ρ\rho is now a free parameter of the model.

A positive ρ\rho lowers σE\sigma_E below σα\sigma_\alpha (Eq. 10 of the revised manuscript), which is exactly the inconsistency you spotted.

In the revised manuscript the ansatz is now a bivariate Gaussian for (αb,log⁡βb)(\alpha_b,\log\beta_b) (Eq. 3), and Table 1 has a ρ\rho column. All population curves are recomputed with ρ\rho. The error in log⁡10\log_{10} survival on Qwen-3B goes from 0.58 in the submission to 0.12 with ρ\rho, and to 0.09 with ρ\rho and the new estimator (point 4). On Qwen-7B-Instruct it goes from 0.26 to 0.21 and 0.09.

  1. Clipping and exponential saddles

We tested the effect of clipping and saw negligible differences. Using the exact clipped mean, ⟨p⟩a=1−eβbTαb/(1+βbT)\langle p\rangle_a=1-e^{\beta_bT\alpha_b}/(1+\beta_bT) for αb<0\alpha_b<0, changes ASR@NN by at most 0.012 for Llama and by at most 0.003 for every other model and every NN (App. C).

On the higher-order moments: mean matching is only used to define αb\alpha_b (Eq. 21). The attack surface at M>1M>1 is computed from the Beta law itself (Eq. 22 and App. B), so its higher moments are those of the fitted Beta and not of the hard-edged law. They are tested by the extrapolation in MM, from M≤40M\le40 to M∼300M\sim300 (Fig. 4b). The Beta is the within-prompt law we actually use, and the hard wall is its motivation.

We agree that the Beta-fit does not directly validate the exponential form. It simply supports that the exponential form is consistent with the data. Some other distribution might be a more accurate depiction of the actual behavior of the model. We rephrased our claims accordingly. We used an exponential to be able to make analytical predictions for the ASR and because it had the clearest physical interpretation. However, the core message that defense is organized through entropic (independent of TT) and enthalpic (proportional to 1/T1/T) channels does not rely on the choice of distributions for the saddles and/or αb,log⁡βb\alpha_b, \log \beta_b. It is tested directly by the temperature experiment (Fig. 5). The modelling choices we made should be seen as a first, most obvious, guess. Not a definitive truth.

  1. Exponential crossover

The revised Fig. 7(c) uses 300 random PP-subsets of the 400 prompts for each PP, so that only PP changes and we don't influence the difficulty directly. Each subset follows the population law until its own hardest prompt takes over, and this break-away moves to larger NN as PP grows. Quantitatively, on Qwen-3B (Granite labels) the median resolution budget Nres=1/min⁡bqbN_{\rm res}=1/\min_b q_b is 52, 65, 75, 85, 111 and 111 for P=10,25,50,100,200,400P=10,25,50,100,200,400, while the extreme-value prediction of the fitted law, which has no floor, gives 49, 62, 72, 84, 96 and 108. A floor qmin⁡>0q_{\min}>0 would instead pin NresN_{\rm res} near 1/qmin⁡1/q_{\min}. This excludes a floor above qmin⁡≈0.009q_{\min}\approx0.009. A lower one cannot be excluded with 400 prompts, which is why we now write that the crossover is "consistent with" a finite-size effect. Any judge will eventually run into the finite-size effect, and a judge which triggers more easily simply runs into it first. With the stricter HarmBench labels the law holds over the whole 10410^4 arm with no crossover (Fig. 7(a-b)), as expected since it predicts Nres≈2×104N_{\rm res}\approx2\times10^4. We have also replaced "fake crossover" by "finite-size crossover" throughout.

  1. Prompt filtering

We agree and have revised our fitting mechanism to address this issue (general response point 2). The five parameters are now fitted by maximizing the marginal likelihood over all prompts, with the per-prompt (αb,βb)(\alpha_b,\beta_b) integrated out (Eq. 22), so that never-jailbroken prompts simply enter as censored observations. The submitted per-prompt moments are kept for comparison in App. C (Table 2). This improved estimator led to lower errors on predicted ASR for all models. The correction is largest for Llama, whose μE\mu_E rises from 2.05 to 2.75, and the comparison between models of Sec. 4 is unchanged.

The authors fit a Beta distribution and show it predicts success rates when they sample more answers per prompt variation. What remains open is whether the proposed exponential barriers explain those rates. The exponential tail adds little evidence for that mechanism. A finite dataset can produce such a tail simply because the hardest prompt eventually accounts for most of the remaining failures. So accurate labels on the hardest prompts matter a lot. For the 20 hardest prompts, human reviewers checked 97 answers flagged by Granite alone and judged only two to be harmful. Judge errors could therefore shift the point where the curve turns exponential. I see value in studying both sampling budgets at different temperatures. Still, I find the paper's novelty and evidence limited and will keep my original score.

Thank you for the follow-up. We respectfully disagree with this assessment, and answer the points below.

Exponential barriers. The paper does not rest on the exponential form, and the revised text does not claim that the Beta fit establishes it. The mechanism we propose is that each augmentation selects a barrier which is then crossed thermally, on top of an entropic bottleneck set by the prompt. Three tests support it beyond the quality of the Beta fit. (i) The barrier can be read from the residual of the augmented prompt alone, before any completion is generated (held-out R2R^2 up to 0.76, App. H). (ii) Moving an augmentation so that this barrier is lowered or raised moves the jailbreak rate from 5.6% to 16.4% or to 2.2%, with a slope of 0.93 nats per standard deviation against 1.03 for the correlation, while the same move in a random direction does nothing (App. I). (iii) The ASR at four unseen temperatures is predicted with an error 3.7 to 7.5 times smaller than pure Arrhenius (Fig. 5). Moreover, the results in NN do not depend on the law of the barriers at all. At M=1M=1 the law of pb,ap_{b,a} within a prompt enters only through its mean, so the extrapolation in NN, the drifting exponent and the crossover rely only on the Gaussian law of the effective barrier EbE_b across prompts.

Exponential tail. We never use the exponential tail as evidence for the mechanism. Sec. 4.3 shows that it is what a finite dataset produces once its hardest prompt carries the remaining failures, which is the statement you make. We also test it: on random subsets of PP prompts the measured resolution budget is 52, 65, 75, 85, 111 and 111 for P=10P=10 to 400400, and the fitted law predicts 49, 62, 72, 84, 96 and 108 with no additional parameter (Fig. 7c).

Labels of the hardest prompts. The judge sets where the crossover happens, and the paper says so (Sec. 4.3 and App. G). A judge which triggers more easily simply runs into the finite-size effect first, but any judge will eventually run into it. So yes, "judge errors" do shift the point where the curve turns exponential. What we provide is a way to measure and predict where this crossover happens for any judge (even a perfect human annotator), namely at Nres=1/min⁡bqbN_{\rm res}=1/\min_b q_b. With HarmBench labels the law predicts Nres≈2×104N_{\rm res}\approx2\times10^4 on Qwen-3B, and indeed there is no crossover over the whole 10410^4 arm (Fig. 7(a-b)).

Novelty and evidence. To our knowledge this is the first model of the full (N,M)(N,M) attack surface and of its temperature dependence. Five numbers fitted at N≤100N\le100 predict the survival at N=104N=10^4 within 0.02–0.09 decades under three label sets, against 0.10–0.29 for the best existing baseline, and they remain predictive on held-out prompts and on a second dataset (App. J). To our knowledge it is also the only model which predicts the ASR at other temperatures zero-shot. Each of its five constants has a physical interpretation, which gives a mental picture of how the defense of an LLM is organized.

Reviewer_o5M2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The work proposes viewing best-of-N jailbreak scaling behavior as a physical model, and suggests the scaling exponent drifts with log(NM) rather than an exact power law. In the model, each prompt has a safety-alignment based barrier, prompt perturbations (N) select a random saddle, and re-sampling the same prompt (M) is an independent attempt over the fixed saddle. The 4-parameter model is able to reproduce the (N,M) attack surface as accurately as an 800-parameter per-prompt fit, is able to predict large-N scaling behavior better than prior power-law fits, and is able to predict ASR at unseen temperatures from just T=1 data.

Critical strengths
  1. Model fits the data better than a single power law - modeling per-prompt rather than in aggregate makes sense. Separating N and M sampling and finite-P hardest prompt takeover nicely separates sources of randomness that prior works overlook.

  2. The experiment to extrapolate the two budgets from a fit/test split is a harder test than curve-fitting a single power law. The microscopic view also makes non-obvious predictions that are then verified in the residual stream: a linear safety hyperplane and saddle heights predictable before generation.

Critical weaknesses
  1. Judge stochasticity is not modeled despite results depending on its labels. Random or correlated mislabels within a completion batch would contaminate near-zero rate jailbreak regions where per-prompt parameters are identified. Safer models would be more impacted as their true jailbreak rates may be close to judge's error floor. Relatedly, the fake crossover claim depends on counting a number of jailbreaks for the hardest prompt, which can be very small for safer models.

  2. Direct experimental support for the "no power law" claim is thin. On Qwen2.5-3B, the proposed model and beta-fit baseline tie, so the extrapolation improvement relies on qwen2.5-7B and the single-model local-exponent plot which mixes the exponent drift with the finite size dataset artifact.

Questions for authors
  1. The extrapolation test uses the same 400 prompts for fit/test so the finite-dataset effects cancel, but does the model extrapolate to new prompts, which would be more relevant to what a defender wants?
  2. Is there any causal evidence for probe results? E.g. does intervening along saddle or safety direction change p as predicted? And is the saddle readable at layers earlier than layer 30/36?
3 comments

Thank you for your specific and testable concerns. We ran every test you suggested.

  1. Judge stochasticity & Fake crossover

Although the constants we fit for our model depend on the judge's labeling, the structure of our proposed modelling does not. To show this we refitted everything with HarmBench labels, and with labels requiring both judges to agree. The barrier level depends on the judge (μE\mu_E = 3.07, 6.53 and 7.56 for Qwen-3B), but the predictive power of the model does not. Fitting at N≤100N\le100 and testing on the 10410^4 arm for 100<N≤104100<N\le10^4 (the cells of Fig. 4a), the error in log⁡10\log_{10} survival is (ours / Feng et al. / power law):

labelsQwen-3BQwen-7B-Instr.
Granite0.09 / 0.22 / 1.300.09 / 0.29 / 0.30
HarmBench0.017 / 0.19 / 0.420.031 / 0.24 / 0.21
both agree0.023 / 0.10 / 0.240.022 / 0.15 / 0.15

A judge false-positive rate can also be included in the fit; it barely moves (μE,σE)(\mu_E,\sigma_E) (App. G).

We also looked at the judges by hand (App. G). In a blind audit of 500 completions of Qwen-3B by a single annotator, 14% of Granite's jailbreaks were labelled harmful, against 79% for HarmBench and 88% when both judges agree. The mechanism holds under all three label sets: the judges' definition of harm is only reflected in the fitted constants of the model. A possible practical follow-up would be to run this modelling with human labels.

On the crossover: under Granite every prompt of Qwen-3B is jailbroken at least 90 times in the 10410^4 arm, so the count of the hardest prompt is not small. The judge does change where the crossover happens. Any judge will eventually run into the finite-size effect, and a judge which triggers more easily simply runs into it first. Nres≈102N_{\rm res}\approx10^2 for Granite, whereas under the stricter HarmBench labels the law holds up to N=104N=10^4 with no crossover (Fig. 7(a-b)).

  1. No power-law

With the improved model and fitting method our method now clearly outperforms the beta-fit baseline on Qwen2.5-3B (see Fig. 4a). We have also added a new test to support the "no power law" claim. For Qwen-7B-Instruct NresN_{res} is larger than our resolution so there is no visible finite-size effect muddying the results. Over N=30,100,300,103,3×103N=30,100,300,10^3,3\times10^3 the exponent keeps rising, as 0.33, 0.49, 0.63, 0.74, 0.89 under Granite and 0.13, 0.24, 0.37, 0.53, 0.65 under HarmBench. Our law predicts 0.13, 0.23, 0.34, 0.48, 0.63 under HarmBench. Under Granite it predicts 0.30, 0.48, 0.68, 0.91, 1.14, which overshoots beyond N=103N=10^3. In comparison, a power law fitted at N≤100N\le100 stays between 0.13 and 0.28 under HarmBench and between 0.30 and 0.46 under Granite.

  1. Extrapolation to new prompts

This was an excellent suggestion, and we added it (App. J). Over 20 random 200/200 splits of the prompts, we fit the law on one half and predict the other. In five of the six (model, labels) cases our model predicts the survival of the unseen half on the 10410^4 arm within 0.02–0.10 decades, against 0.11–0.23 for the Beta-fit baseline (Table 13). The exception is Qwen-3B with Granite labels (0.46 ± 0.43 against 0.59), where the tail of a 200-prompt half is set by one or two prompts.

  1. Causal probes

We now report (App. H) the held-out R2R^2 of log⁡p^b,a\log\hat p_{b,a} from the prompt's last-token residual at every layer, with cross-validation grouped by prompt. In every model the probe is at zero at the embedding layer, so it is not lexical, and it peaks at 80–95% depth. For Qwen-3B it goes 0.24 (layer 4), 0.32 (12), 0.41 (20), 0.56 (24), 0.67 (31 of 36), and for Llama 0.32 (12), 0.64 (16), 0.76 (29 of 32).

We added two interventions on Qwen-3B (new App. I). First, we push the residual stream at layer 30 along the safety direction, along the saddle direction, or along a random direction of the same norm. Both directions move the jailbreak rate up and down, whereas a random direction does not. Second, in our model the saddle is selected by the augmentation, so we act on the augmentation itself. We move its noise along the direction which lowers (or raises) the saddle read by the probe while keeping its norm, so that it remains as probable under the kernel. The jailbreak rate rises from 5.6% to 16.4% when the saddle is lowered and falls to 2.2% when the saddle is raised. The same displacement in a random direction does nothing. The effect decreases with temperature. These numbers use Granite labels. With HarmBench labels the directed augmentation acts in the same way (from 0.73% to 0.14% when the saddle is raised, and to 1.8% for the largest lowering).

  1. We have also aimed to improve the presentation of the paper. We added a notation table (App. K), we now define TT and the five parameters right after Eq. 1, and we made targeted fixes throughout the paper to improve readability. If the reviewer has any specific presentation-related concerns which we might be able to improve we'd be happy to implement them.

Thank you for your detailed response and revisions.

Most of my concerns and questions have been addressed, so I raise my score.

Thank you!

Reviewer_T7vCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Best-of-N jailbreaking and proposes a physics-inspired model to characterize the attack behavior under different augmentation and completion budgets. The model introduces a small number of interpretable parameters and is evaluated on several open-source language models. The paper further studies extrapolation, temperature dependence, and different attack settings.

Critical strengths
  1. The paper is generally well-written, and the motivation is interesting.

  2. The reviewer appreciates the relatively extensive experimental validation, including extrapolation and several robustness analyses.

Critical weaknesses
  1. Based on the reviewer’s understanding, the mechanistic interpretation may still have somewhat limited validation across models. The residual-stream analysis used to support the physical picture seems to be mainly conducted on one model and one layer. It is therefore not fully clear whether the same internal interpretation holds for the other model families evaluated in the paper.

  2. The generality across harmful-prompt distributions is also somewhat unclear. The experiments are mainly based on a single adversarial dataset with a relatively limited number of prompts. Since some of the paper’s conclusions depend on the tail of the prompt-level barrier distribution, the reviewer wonders whether the same observations would hold on substantially different safety benchmarks or prompt distributions.

  3. The paper argues that the general structure is robust across augmentation kernels. However, based on the reviewer’s understanding, the kernel comparison is relatively limited and mainly considers embedding noise and one character-level perturbation setting. It is therefore still somewhat unclear how broadly this universality claim extends to other forms of independently sampled prompt perturbations.

Questions for authors

Please see the weaknesses above.

3 comments

Thank you for your positive assessment of our work and for the concrete suggestions. We address each one below.

  1. Based on the reviewer’s understanding, the mechanistic interpretation may still have somewhat limited validation across models. The residual-stream analysis used to support the physical picture seems to be mainly conducted on one model and one layer. It is therefore not fully clear whether the same internal interpretation holds for the other model families evaluated in the paper.

We ran the saddle probe on all five models with a full layer sweep (App. H). Grids with few completions per cell measure log⁡p^b,a\log\hat p_{b,a} noisily, so for each model we also estimate the binomial noise ceiling, i.e. the fraction of the variance that any predictor of pp could explain (a split-half estimate).

modelbest layerheld-out R2R^2Spearmannoise ceilingR2R^2/ceiling
Qwen2.5-3B-Instruct31/360.670.790.900.74
Qwen2.5-7B-Instruct23/280.700.780.960.73
Llama-3.1-8B-Instruct29/320.760.820.940.81
Gemma-2-9B-it35/420.190.400.440.43
Qwen2.5-7B (base)26/280.400.570.610.66

For the two Qwen-Instruct models and Llama the probe recovers 73–81% of the explainable variance. Gemma is the weak case: its within-prompt variance of log⁡p^\log\hat p is half of Llama's, so most of it is binomial noise, and even relative to its ceiling it is the lowest. This is consistent with our modelling which showed that Gemma has nearly no energetic defense (and hence no saddles) and relies almost exclusively on an entropic bottleneck.

In every model the probe is at zero at the embedding layer, so it is not lexical, and it peaks at 80–95% depth.

  1. The generality across harmful-prompt distributions is also somewhat unclear. The experiments are mainly based on a single adversarial dataset with a relatively limited number of prompts. Since some of the paper’s conclusions depend on the tail of the prompt-level barrier distribution, the reviewer wonders whether the same observations would hold on substantially different safety benchmarks or prompt distributions.

We agree that this was a limitation. To mitigate it, we ran a new dataset: the 200 HarmBench standard behaviours on Qwen2.5-3B-Instruct (σ=0.048\sigma=0.048), as a 200×50×100200\times50\times100 grid plus an independent 200×1950200\times1950 experiment, and scored them under all three label sets (Granite / HarmBench / both). The results are in App. J.

  • (i) The structure of the model holds. The KS pp-values for the standardized α^b\hat\alpha_b, log⁡β^b\log\hat\beta_b and EbE_b are 0.48, 0.34, 0.39 / 0.35, 0.48, 0.21 / 0.39, 0.69, 0.93, and the (N,M)(N,M) surface is reproduced with an MAE of 0.033 / 0.010 / 0.007, similar to an AdvBench grid of the same size (0.030 / 0.010 / 0.010).
  • (ii) Categories. Under Granite the mean barrier differs across HarmBench's six semantic categories (ANOVA p<10−11p<10^{-11}, with means from 2.6 to 3.8), so the across-prompt law is a mild mixture: the s.d. of EbE_b is 0.77 overall and 0.66 within categories, where the residuals are Gaussian (KS p=0.28p=0.28). Under the stricter labels the category effect is not significant (p=0.07p=0.07 and 0.18).
  • (iii) Extrapolation from N≤50N\le50 to the independent arm. Under Granite our law misses by 0.11 decades, against 0.19 for the Beta law and 0.78 for a power law. Under the stricter labels this weaker attack cannot discriminate between methods: 86 and 135 of the 200 prompts are never jailbroken in 1950 augmentations, and all three laws are within 0.04 decades (ours 0.005 and 0.004).

The random-subset and held-out-prompt tests (see our replies to 4vZ5 and o5M2) also address similar concerns.

  1. The paper argues that the general structure is robust across augmentation kernels. However, based on the reviewer’s understanding, the kernel comparison is relatively limited and mainly considers embedding noise and one character-level perturbation setting. It is therefore still somewhat unclear how broadly this universality claim extends to other forms of independently sampled prompt perturbations.

We agree that two kernels do not justify the word "universal", and the revision now restricts the claim to the kernels we tested (App. F). Across the seven (kernel, strength) configurations we have on Qwen2.5-3B-Instruct, namely the character kernel and the embedding kernel at six values of σ\sigma, Gaussianity of α^b\hat\alpha_b and log⁡β^b\log\hat\beta_b is never rejected.

A third kernel that changes the meaning of the prompt, such as LLM paraphrases or random-token insertion, is feasible for the camera-ready version, but we could not run it during the discussion period. Iterative attacks such as GCG or PAIR remain out of scope, since their augmentations are not independent.

Thanks for the authors' rebuttal, after reading it the reviewer finds that most of the concerns have been resolved and thus keeps the positive score. The reviewer would very much like to see additional experiments with more kernels in the camera-ready version.

Thank you!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General reply to common concerns raised by all reviewers.

General response (to all reviewers)

We would like to take a moment to thank the reviewers for their careful feedback. We think that the comments provided by the reviewers were relevant, useful and have significantly increased the quality of the submission. We provide below general answers to common points raised by multiple reviewers and we will provide more in-depth individual comments to each referee's point subsequently. We have also submitted an updated manuscript with all changes marked in blue, and changed figures are framed in blue.

  1. Model specification (4vZ5). The ansatz omitted the correlation ρ\rho between αb\alpha_b and log⁡βb\log\beta_b (Eq. 3 in the revised manuscript). We fully agree, thank you for pointing it out. With our previous per-prompt estimates the correlation looked negligible on the larger models (Qwen-7B-Instruct, Gemma and Llama), and we erroneously assumed that to be the case for Qwen-3B as well. The revised fit (point 2) shows that it is in fact not negligible for any model, with ρ\rho between −0.52-0.52 and +0.48+0.48 depending on the model. We have restated the microscopic model as a five-parameter model: a two-dimensional Gaussian with cross-correlation ρ\rho. Adding this term reduces the extrapolation error on Qwen-3B by about 5x (from 0.58 to 0.12 decades beyond the fit window, and to 0.09 with the new fit of point 2), which also addresses o5M2's comment about Qwen-3B and the baseline Beta fit tying. With this improved description our estimator clearly outperforms the Beta fit of Feng et al. (see the revised Fig. 4a).

  2. Estimation (4vZ5, o5M2). We replaced our previous two-step fit with a hierarchical marginal-likelihood fit that keeps never-jailbroken prompts as censored observations and can include a judge false-positive rate (Eq. 22 and App. C). It makes (μE,σE)(\mu_E,\sigma_E) insensitive to filtering. With the new fit, Llama's predicted ASR is within 0.04 of the measured one.

  3. Judges (o5M2). Every fit and test on the two Qwen-Instruct models is repeated with HarmBench labels and with labels requiring both judges to agree. With the stricter labels the hierarchical law fitted at N≤100N\le100 predicts the 10410^4-augmentation survival to N=104N=10^4 within 0.017–0.031 decades, against 0.10–0.24 for the Beta law and 0.15–0.42 for a power law (Table 7). We've also done a blind human audit (App. G; details in our reply to o5M2). However, we'd like to point out that the purpose of the paper is to propose this new interpretable mechanism behind BoN jailbreaking. The mechanism holds under all three label sets we tested: the judge choice mainly influences the definition of what is considered 'safe' or 'unsafe', and hence is reflected in the fitting constants of our model.

  4. New tests. To answer concerns raised by the reviewers we ran some new tests. We checked the finite-size effect using random subsets (our conclusion still holds), checked the extrapolation to held-out prompts (our model is still predictive and better than the Beta-law fit), and repeated the analysis on a second dataset, the HarmBench standard behaviors (App. J). We also added interventions on the safety and saddle directions (App. I and reply to o5M2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.