acceptodds
Under review as a conference paper at ICLR 2027

Refusal Is Not Robustness: Measuring Harmful Reachability in Autoregressive Language Models

Abstract

LLM safety evaluations via jailbreaking typically index on attack success rate (ASR): whether a generated response violates safeguards. However, ASR only evaluates sampled outputs and provides little information on the underlying generation process. Autoregressive language models define distributions over possible outputs, and a sampled refusal can coexist with substantial probability mass on harmful generation paths. We study this hidden structure through bounded continuation analysis (BCA), which models autoregressive generation as a stochastic process over token prefixes and computes the finite-horizon probability that generation reaches a harmful continuation. BCA constructs a bounded continuation tree, labels terminal continuations with a semantic harm evaluator, and estimates harmful reachability using exact probabilistic model checking. BCA complements ASR by measuring how much harmful behavior remains accessible beneath a sampled safe response. This makes it useful for distinguishing models that look identical under measured ASR values, and allocating red-team effort toward regions of the generation process with greater hidden risk. Across four instruction-tuned LLMs and 100 JailbreakBench behaviors, we show that BCA effectively quantifies hidden harmful behavior and provides a useful signal to guide red teaming.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
AI Safety, Red Teaming, Formal Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_d1BPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors study whether an observed refusal adequately characterizes the safety of an autoregressive language model’s continuation distribution. The paper introduces Bounded Continuation Analysis (BCA), which constructs a continuation tree, evaluates terminal continuations with a semantic harm judge, and aggregates their probabilities to quantify bounded harmful reachability. The authors further introduce Generative-Attack Susceptibility (GAS), which tracks the maximum bounded harmful mass among prompts encountered within an attack budget. Overall, the paper presents a potentially useful perspective on distribution-aware safety evaluation.

Critical strengths
  1. The paper addresses an important limitation of evaluating safety from a single decoded response. A refusal under greedy decoding does not establish that other likely continuations are safe, and examining the probability assigned to alternative responses is a well-motivated direction. The distinction between observed refusal and distributional vulnerability is relevant to both model auditing and the allocation of red-teaming effort.
  2. The tree-based approach is straightforward and well motivated, providing an intuitive way to explore alternative continuations and quantify their harmful probability mass.
Critical weaknesses
  1. The advantage of BCA over direct sampling remains unclear. Could the authors compare it with batch generation under different temperature or top-kk settings, using matched computational budgets, to establish its practical benefit for jailbreak risk evaluation?

  2. It is unclear whether eight-token prefixes reliably indicate harmful completed responses. Could the authors clarify the evaluator inputs and validate prefix judgments against completed rollouts, including sensitivity to the generation horizon?

  3. The methodological novelty appears limited. More comprehensive sensitivity analyses of α\alpha, kk, and LL, together with evaluation on an additional benchmark, would strengthen the empirical contribution.

Questions for authors

My main questions are incorporated into the weaknesses above. The most consequential clarifications are what the eight-token leaf labels represent, how much probability mass the bounded trees retain, and whether BCA improves coverage over repeated judge-guided search at the same total cost. Reconciled experimental counts and independently evaluated final selections would further strengthen confidence in the findings.

1 comment
AuthorsOpenReview ↗

Response to R3

1. Comparison with direct sampling

We agree that direct sampling is an important baseline, but emphasize that BCA is primarily intended as a formal diagnostic tool for internal model safety evaluation rather than a more efficient sampling strategy. Unlike Monte Carlo estimation, BCA computes exact harmful reachability within a bounded continuation tree, providing a deterministic lower bound without sampling variance. This is particularly useful for identifying low-probability harmful continuations that may be missed by finite sampling. We acknowledge that matched-budget comparisons across decoding temperatures and top-p settings would further clarify its computational trade-offs, although these settings also change the underlying generation distribution being evaluated.

2. Reliability of eight-token prefix evaluations

We acknowledge that evaluating harmfulness from short prefixes introduces uncertainty, as an eight-token continuation may not fully capture the eventual behavior of the model. Our current approach evaluates bounded continuations using a semantic harm judge, but does not establish that these judgments always agree with completed responses. Nevertheless, Table 2 provides empirical validation against independently sampled full generations, demonstrating positive correlations of 0.534–0.733 between BCA scores and realized harmful outcomes. We will clarify this distinction and discuss the importance of evaluating longer horizons and prefix-to-completion consistency.

3. Methodological novelty and sensitivity analysis

While our work builds on the bounded probabilistic model-checking formulation introduced by LLMChecker, our contributions extend beyond applying an existing technique. Specifically, we introduce GPU-batched continuation analysis, extend model checking to semantic harmful reachability, and formulate GAS to characterize susceptibility under adaptive adversarial search. We further demonstrate that BCA reveals vulnerabilities hidden by saturated ASR measurements and identifies failures complementary to judge-guided search.

Reviewer_uzxmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new metric for measuring jailbreak effectiveness: BCA. This differs from the single-sample ASR computation in constructing a limited continuation tree for a given malicious input, labeling leaves with a semantic harm evaluator, and estimating harmfulness by checking the cumulative probability of reaching harmful leaves. BCA is also extended for adaptive jailbreaking methods into GAS, which is defined as the maximum BCA obtained over the set of prompts generated during the jailbreaking process. The paper shows some evidence of correlation between GAS and the potential for a model to yield a high ASR after being targeted by an adaptive jailbreaking approach, as well as BCA and ASR computed using many samples. BCA also shows some potential as a target optimized for by adaptive jailbreak attacks.

Critical strengths
  • BCA appears to provide a more robust alternative to ASR for assessing jailbreak efficacy, as it captures the degree to which a model is still vulnerable to an attack even if it's most probable generation results in a refusal. Some evidence is presented for the correlation of BCA with ASR computed over many samples.
  • The computation of BCA itself is intuitive, essentially calculating the probability of a beam search starting from the adversarial prompt resulting in harmful completions.
Critical weaknesses
  • The main weakness of this work is that it is unclear what advantages BCA presents over empirical ASR. From a practical standpoint, jailbreakers are only interested in the probability of getting refusal when targeting a model with an attack, which seems more aligned to empirical ASR over a large number of samples. No analysis is provided over the computational overhead of BCA, and it does not prove to be a significantly more effective target for adaptive strategies to optimize for compared to ASR, further limiting it's utility.
  • I am also unsure about the strength of correlation between BCA, GAS and the model's actual vulnerability to attacks when being prompted. For example, in Table 2, the GAS@18 values seem to differ significantly even against the ASR@18, making deltas in GAS@18 harder to interpret from a practical standpoint.
Questions for authors
  • Can the authors elaborate on the complementarity between BCA and judge-based approaches? I'm not sure that an union is the best metric to analyze for those experiments, as ultimately the metric to care about is the average quality of the attacks being generated by these approaches. How would these methods be unified from a practical standpoint?
  • How does BCA compare to just sampling more data points for ASR, in terms of efficiency and alignment with human preferences?
  • The authors mention that it could be possible for the BCA computation to stop without hitting the probability mass cap. Does this actually happen in any of their experiments?
1 comment
AuthorsOpenReview ↗

Response to R2

Responses to Weaknesses

  • Advantages of BCA over empirical ASR: We would like to clarify that BCA is primarily intended as a diagnostic tool for model developers and internal safety teams, rather than as a standalone optimization objective for external jailbreak attackers. While empirical ASR measures the success of observed attacks, BCA provides a formal characterization of harmful probability mass within the model's continuation distribution, revealing latent vulnerabilities even when sampled responses consistently refuse. This enables developers with access to model probabilities to identify fragile safety behaviors, prioritize adversarial testing, and improve safety alignment before these vulnerabilities can be exploited. Our adaptive search experiments demonstrate an additional application of BCA, rather than its primary purpose. We acknowledge the computational overhead and will clarify that the intended benefit is deeper safety analysis rather than necessarily outperforming empirical ASR in attack efficiency.
  • Correlation between GAS and actual vulnerability: GAS and ASR measure different quantities and are therefore not expected to agree numerically. Specifically, GAS@18 represents the maximum bounded harmful probability across 18 templates, whereas ASR@18 measures the fraction of behaviors for which at least one template produces a harmful response. Consequently, differences in GAS can remain meaningful even when ASR saturates. More importantly, Table 2 directly validates BCA against independently sampled harmful generations, demonstrating positive Spearman correlations of 0.534–0.733 and improvements of 2.56×–7.43× in realized ASR when selecting templates using BCA.

Responses to Questions

  • Complementarity and practical integration with judge-based approaches: We agree that union coverage alone does not characterize average attack quality. Our goal with this experiment is to demonstrate that BCA and judge-guided search discover partially disjoint failures, rather than establish that BCA is a superior standalone attack objective. Table 3 further demonstrates a practical integration: using BCA for candidate exploration and an outcome-based judge for final selection recovers deployment ASR substantially compared to BCA-only selection. This suggests a hybrid approach where BCA guides exploration while judge-based evaluation determines which attacks are ultimately deployed.
  • Comparison with additional ASR sampling and human preferences: Monte Carlo sampling and BCA offer different guarantees. Sampling provides a statistical estimate of harmful-generation probability, whereas BCA deterministically aggregates probability over a bounded continuation tree, potentially identifying low-probability harmful paths that finite sampling misses. Our current results establish predictive agreement with independently sampled outcomes. Furthermore, because semantic harm evaluation relies on learned judges, BCA's alignment with human preferences depends on evaluator quality.
  • Whether the probability mass cap is reached: Our implementation retains the smallest set of high-probability tokens satisfying the cumulative threshold (\alpha), subject to the maximum branching factor (k). Consequently, when the top-(k) tokens do not cover (\alpha) probability mass, the branching constraint takes precedence and the remaining probability mass is discarded. This behavior is explicitly accounted for in our bounded formulation. In our experiments, it occasionally prevents the retained transitions from reaching the cumulative probability threshold (\alpha). However, we observe that the relative ordering of model susceptibility remains consistent even in these cases, suggesting that the bounded approximation continues to provide a meaningful comparative signal.
Reviewer_nRotOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that evaluating LLM jailbreak robustness via Attack Success Rate (ASR) on single decoded outputs obscures "distributional fragility": the probability mass assigned to harmful continuations just beneath a sampled refusal. Building on LLMChecker (Gross et al., 2025), the authors propose Bounded Continuation Analysis (BCA), which constructs an α\alpha-kk bounded generation tree up to horizon LL (L=8,k=2,α=0.99L=8, k=2, \alpha=0.99) and evaluates leaf prefixes with an LLM judge to compute bounded harmful reachability. They further define Generative-Attack Susceptibility (GAS) to track the maximum BCA mass over an attacker's query budget (using static templates, PAIR, and TAP). Experiments across four open-weight 1.5B–8B models on JailbreakBench show that BCA differentiates models with 0% greedy ASR, correlates with 100-sample empirical ASR, and uncovers complementary jailbreaks when used as a pruning signal in TAP and PAIR.

Critical strengths
  1. Intuitive motivation: Moving beyond point-estimate greedy ASR to evaluate the local continuation distribution around refusal boundaries addresses a well-known limitation in current safety benchmarking.

  2. Batched implementation over LLMChecker: Adapting bounded model checking from per-node recursive queries to depth-wise GPU-batched expansion makes tree construction practically viable during iterative red-teaming loops.

  3. Complementary exploration in tree search: Demonstrating that distribution-guided pruning (BCA) and outcome-guided pruning explore partially disjoint regions of the prompt space is an interesting empirical observation for automated red-teaming.

Critical weaknesses
  1. The relationship between bounded harmful reachability and actual harmful-generation probability is insufficiently characterized:

BCA computes harmful probability mass within a truncated continuation tree, rather than over the full autoregressive distribution. The paper defines the retained token set using an alpha-k bounding scheme, but in practice uses alpha = 0.99, k = 2, and L = 8. When the branching cap prevents retention of 99% of the outgoing probability mass, the stated alpha-based mass guarantee no longer applies. Consequently, the proportion of total continuation probability retained by BCA may vary substantially across prompts and models.

This raises an important interpretability concern: differences in BCA scores could reflect differences in the fraction of probability mass retained, rather than only differences in harmful-generation probability. For example, a model with a highly concentrated next-token distribution may have substantially more of its continuation probability retained by top-2 expansion than a model with a diffuse distribution. Comparing their unnormalized harmful masses could then conflate model safety with distribution concentration. Although BCA is explicitly presented as a bounded measure, the paper frequently interprets larger GAS values as evidence of greater underlying harmful susceptibility. This interpretation requires additional validation.

  1. The practical benefit over simpler sampling-based alternatives is unclear:

One important result is that BCA correlates with empirical ASR measured through repeated independent sampling. However, this also raises a natural baseline question: how much additional value does BCA provide over estimating harmful-generation probability directly through stochastic sampling? A straightforward alternative would be to sample multiple completions from the target model and calculate the fraction classified as harmful. While such an estimator has sampling variance, it is unbiased for the harmful probability under the corresponding decoding distribution, assuming a fixed evaluator and sampling configuration. In contrast, BCA deterministically computes probability mass over a truncated continuation tree, introducing approximation bias from discarded branches and finite generation length. The paper does not adequately quantify the trade-off between these alternatives in accuracy, cost, and ranking quality.

In particular, it would be helpful to compare BCA against Monte Carlo estimation at matched inference budgets, measuring correlation with empirical ASR, ability to identify high-risk prompts, and downstream effectiveness in adaptive attack search. Without this comparison, it remains unclear whether bounded probabilistic model checking is a practically advantageous method for estimating the proposed safety metric.

  1. Numerical inconsistencies in the adaptive red-teaming results:

There seems to be inconsistencies between the reported success rates and the mutually exclusive outcome decompositions in Tables 3 and 4. For example, Table 3(b) reports that, for Llama-3.1-8B, both search methods succeed in 37.3% of runs and only BCA-guided search succeeds in 21.6%. This implies that BCA-guided search succeeds in 58.9% of runs, while Table 3(a) reports a deployment ASR of 58.3%. The discrepancy is larger in Table 4: For Llama-3.1-8B, the paired outcome decomposition implies a judge-guided success rate of 38.0% + 16.0% = 54.0%, whereas Table 4(a) reports 56.9%. Similarly, the implied BCA-guided success rate is 50.0%, rather than 51.7%. Furthermore, Table 3(b) specifies N = 120 paired runs per model, but several reported percentages in the Llama-3.1-8B column cannot arise from 120 binary outcomes when rounded to one decimal place. These discrepancies are larger than ordinary rounding errors and require clarification. They are particularly relevant because the complementarity between outcome-guided and BCA-guided search is one of the paper's primary empirical contributions.

  1. Accuracy and consistency of the literature review:

The manuscript contains a few issues suggesting that the related-work discussion and bibliography require more careful verification. For example, Wei et al. (2023a, 2023b) and Souly et al. (2024a, 2024b) are duplicated references to the same respective publications. More substantively, Section 5 characterizes logprob-targeted attacks (Andriushchenko et al., 2025) as bypassing search entirely, although adaptive search over adversarial suffixes is central to that method. The discussion of shallow safety alignment (Qi et al., 2025) also uses unnecessarily absolute language by claiming that safety training modifies the initial response prefix exclusively. These issues do not invalidate the proposed method, but they raise concerns about the precision of the literature review, particularly where prior work is used to motivate the novelty and technical contributions of BCA.

Questions for authors
  1. How sensitive are the model rankings and GAS measurements to changes in k, L, and alpha?

  2. How does BCA compare with Monte Carlo estimation of harmful-generation probability under matched computational budgets?

  3. To what extent does BCA predict attack success when the attacker cannot control decoding randomness or directly force alternative model continuations?

1 comment
AuthorsOpenReview ↗

Response to R1

Response to Weaknesses

  • Bounded harmful reachability and actual harmful probability: We agree that variation in retained probability mass can affect cross-model comparisons, particularly when next-token distributions differ in concentration. While BCA computes a valid lower bound on finite-horizon harmful probability, a larger BCA score does not necessarily imply greater true harmful probability than another model with a smaller score. We will clarify this limitation and report the retained probability mass alongside BCA scores to distinguish distribution coverage from harmful reachability.
  • Comparison with sampling-based alternatives: We agree that Monte Carlo sampling is an important baseline, but BCA offers a fundamentally different guarantee through formal probabilistic model checking. Rather than statistically estimating harmful-generation probability from sampled completions, BCA computes the exact harmful reachability probability within the constructed bounded DTMC, providing a deterministic, sampling-variance-free lower bound on finite-horizon harmful probability. This enables formally grounded analysis of reachable harmful behaviors, including low-probability continuations that finite sampling may miss. We do not claim that BCA is necessarily more computationally efficient or accurate at estimating the full distribution, but emphasize that its primary contribution is providing a formally characterized measure of harmful reachability rather than another empirical estimator of ASR.
  • Numerical inconsistencies in Tables 3 and 4: We thank the reviewer for identifying these inconsistencies. Upon examining the reported percentages, we find that the Llama-3.1-8B outcome decomposition in Table 3(b) is consistent with 102 paired runs rather than the stated overarching 120, while Table 4(b) is consistent with 100 runs. We will clarify the differences in evaluation subsets, and correct the tables to ensure consistency between marginal success rates and paired outcome decompositions.
  • Accuracy of the literature review: We thank the reviewer for identifying these issues. We agree that the duplicated Wei et al. and Souly et al. references should be consolidated, and that our characterization of prior work requires greater precision. In particular, Andriushchenko et al. employ adaptive search rather than bypassing search entirely, and the discussion of shallow safety alignment should not imply that safety training exclusively modifies the initial response tokens. We will correct these statements and refine the related-work discussion to distinguish our contribution from existing methods more accurately.

Responses to Questions

  • Sensitivity to (k), (L), and (\alpha): BCA's measurements depend on the chosen bounds, and we currently use fixed parameters ((\alpha=0.99), (k=2), (L=8)) across all models to ensure a consistent evaluation protocol. Increasing (k) or (L) expands the explored continuation space, potentially revealing additional harmful mass. We plan to run additional ablations to answer these questions.
  • Comparison with Monte Carlo estimation: BCA and Monte Carlo sampling provide complementary forms of analysis. Monte Carlo statistically estimates harmful-generation probability, whereas BCA uses probabilistic model checking to compute exact harmful reachability within a bounded continuation tree, providing a deterministic lower bound without sampling variance. This formal characterization is particularly useful for analyzing low-p probability harmful continuations that finite sampling may miss.
  • Control by attackers: BCA does not assume that attackers can directly control token-level decoding or force alternative continuations. Instead, it measures harmful probability mass under the model's own generation distribution and uses this information to guide prompt-level adversarial search. Our experiments demonstrate that BCA correlates with independently sampled harmful generations (Spearman's (\rho=0.534)–(0.733)) and improves failure discovery when combined with outcome-guided search. This to ensure that model can be analysed and patched but not to help adversaries exploit this information.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.