acceptodds
Under review as a conference paper at ICLR 2027

Breaking the Refusal Barrier: Stochastic Discovery of Residual Unsafe Behaviors in Aligned Language Models

Abstract

LLM safety alignment often creates refusal modes: dominant, high-probability regions of the output space that steer generation toward refusals, while residual unsafe behaviors persist in low-probability regions. Alignment suppresses harmful responses but need not eliminate these underlying behaviors, raising a practical question for iterative safety hardening: how can residual unsafe behaviors be efficiently discovered and converted into safety-training data? We formulate this as stochastic exploration rather than adversarial prompt optimization, showing that random input perturbations redistribute the model's response distribution away from dominant refusal modes and toward low-probability unsafe modes. Building on this, we introduce SEEK (Stochastic Exploration to Elicit residual-behavior Knowledge), a simple inference-only framework that repeatedly samples perturbed inputs and responses to discover diverse residual unsafe behaviors, without gradient optimization, semantic prompt engineering, or auxiliary attacker LLMs. We cast discovery as a coverage objective and show that coverage increases monotonically with the sampling budget, asymptotically recovering all reachable unsafe behaviors. Across five safety-tuned LLMs and two benchmarks, SEEK's discovery matches or exceeds white-box optimization at larger budgets while using one to two orders of magnitude less compute, and consistently exceeds more efficient optimization-, embedding-, transfer-, and attacker-LLM-based methods. Finally, fine-tuning on SEEK-discovered failures substantially improves robustness, cutting attack success from to at most across optimization-, embedding-, transfer- and attacker-LLM-based attacks, while preserving general task performance. Stochastic exploration thus offers a simple, efficient alternative to adversarial prompt optimization for residual-behavior discovery and safety-data generation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, LLM Safety, Alignment, Negative-Response Discovery, Stochastic Exploration, Red Teaming, Safety Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Mc4rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SEEK, an inference-only framework for discovering residual unsafe behaviors in safety-aligned language models through stochastic input perturbations and repeated response sampling. Its central premise is that random perturbations can make unsafe responses more accessible despite dominant refusal behavior. SEEK collects unsafe responses for safety training, emphasizing behavioral coverage rather than finding a single successful jailbreak. The paper establishes monotonic coverage growth and an expected coverage advantage of resampling perturbations over reusing a fixed perturbation under its stated assumptions. Experiments across five models and two benchmarks show strong discovery performance at larger sampling budgets and favorable computational efficiency relative to optimization-based baselines. Finally, DPO fine-tuning on SEEK-generated preference data improves resistance to multiple attack families on Qwen2.5-7B, while maintaining accuracy on an MMLU subset.

Critical strengths
  1. Practical simplicity and strong discovery performance. SEEK requires neither gradient optimization nor an auxiliary attacker model, yet achieves high discovery rates across five models and two benchmarks. This makes it a potentially useful baseline for generating safety-training data.

  2. Demonstrated downstream safety benefits. The paper evaluates whether discovered failures improve robustness against structurally different attacks, connecting discovery to a practical safety objective. The observed improvements make the approach more consequential than a method evaluated solely through jailbreak success.

Critical weaknesses
  1. The incremental contribution over existing stochastic methods is insufficiently isolated. The paper lacks a matched-cost comparison in which BoN also retains all failures and supplies equally sized training datasets. Consequently, it remains unclear whether SEEK offers a distinct advantage beyond different perturbations and continued collection.

  2. The safety-hardening gains are not clearly attributable to SEEK-generated data. Training also includes affirmative-prefix augmentation, without controls separating its contribution from DPO and the choice of negative-response source. Likewise, the one-versus-twenty-response comparison changes both dataset size and diversity.

  3. The harmfulness evaluation lacks independent validation. A single automated classifier determines unsafe outcomes, without reported human auditing; the second example in Table 8 does not clearly fulfill the harmful request. This uncertainty matters particularly for success rates based on finding at least one classified failure among 1,024 generations.

Questions for authors
  1. Are matched-cost, matched-data-size safety-tuning comparisons with BoN or other negative-response sources available, and was affirmative-prefix augmentation ablated?

  2. Was any human validation of the harmfulness labels performed, and what supports classifying the second example in Table 8 as a successful jailbreak?

  3. How were the fractional ASR values computed from 100 prompts, and were attacks regenerated against the fine-tuned model? Please also reconcile the differing HarmBench averages between Table 1 and Appendix H.

1 comment

The incremental contribution over existing stochastic methods

We respectfully disagree with the characterization that SEEK offers only an incremental variation of existing stochastic attack methods. This interpretation overlooks the fundamental difference in the objectives of the two approaches. BoN is primarily designed as a jailbreak attack methodology: its objective is to find a successful harmful response, and its evaluation is centered on attack success. Accordingly, its perturbations are deliberately constrained to preserve human intelligibility, and its conventional first-success criterion does not explicitly optimize the collection of diverse residual failures.

In contrast, SEEK introduces a coverage-driven negative-response mining framework for safety hardening. Its objective is not merely to establish whether a prompt can be jailbroken, but to systematically discover and retain a diverse collection of residual unsafe behaviors that can provide richer negative supervision for subsequent safety tuning. This difference also motivates SEEK's broader perturbation distribution, which is not constrained to produce human-intelligible prompts, and its coverage-based formulation that explicitly values collecting multiple distinct failures rather than identifying a single successful attack.

Our experiments demonstrate why this distinction matters. At a generation budget of 1024, retaining all unsafe responses discovered by SEEK achieves more than 13 percentage points higher behavioral coverage than retaining only the first unsafe response per prompt (Appendix I). More importantly, the collected failures provide effective supervision for safety hardening: DPO training on SEEK-discovered responses reduces vulnerability not only to SEEK but also to distinct attack families, including GCG, ASETF, PiF, TAP, and PAIR. These results demonstrate that the value of SEEK extends beyond attack success to discovering residual behaviors that generalize across attack mechanisms and can be used to improve model robustness.

We agree that a matched-cost comparison in which BoN also retains all discovered failures and supplies an equally sized training dataset would provide an additional controlled comparison. However, such a comparison would evaluate a coverage-oriented adaptation of BoN, rather than its conventional attack objective. We view the distinction as important: SEEK explicitly formulates and evaluates stochastic exploration as a negative-response mining problem, with behavioral coverage and downstream safety hardening as its central objectives. The contribution is therefore not simply continued collection under a different perturbation strategy, but the formulation, analysis, and empirical demonstration of a coverage-driven discovery framework for safety hardening. We will clarify this distinction and the scope of our comparison in the revised manuscript.

The safety-hardening gains are not clearly attributable to SEEK-generated data

We agree that the current experiments do not fully isolate the contributions of affirmative-prefix augmentation, DPO, and the negative-response source. We plan to conduct controlled ablations with matched dataset sizes and training configurations to disentangle these effects, including the contribution of response diversity.

The harmfulness evaluation lacks independent validation

We have evaluated several ASR using an alternative judge, observing differences of mostly approximately ±5 percentage points, and found only marginal changes after denoising responses. We will add these results in future.

Reviewer_c4q6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper treats the discovery of residual unsafe behaviors, for use as safety-training data, as stochastic exploration rather than adversarial optimization. SEEK randomly shuffles prompt words and appends a random token, resamples a fresh perturbation for every generation, and keeps every response a classifier judges unsafe. A simple coverage analysis shows that discovery is monotone in the budget and that resampling dominates a fixed perturbation. On five aligned models, SEEK reaches about 96% ASR at N=1024N=1024, slightly above GCG at far lower compute. DPO on SEEK negatives reduces the success of several attack families on Qwen2.5-7B-Instruct while preserving MMLU.

Critical strengths
  • Practical and cheap. SEEK needs no gradients and no attacker LLM, parallelizes trivially, and has a large compute advantage over GCG.
  • Coverage is the right objective for data generation. Appendix I shows that keeping all failures covers about 13 points more behavior clusters than stopping at the first success.
Critical weaknesses
  • Limited novelty over BoN. The paper itself states that BoN differs from SEEK only in the perturbation. The theory (monotonicity of 1−(1−p)N1-(1-p)^N and Jensen's inequality) holds for essentially any sampler, so the contribution rests mainly on the empirical results.
  • The main metric is not validated for this setting. ASR at N=1024N=1024 counts a prompt as broken if any of 1024 samples is judged unsafe, so even a small per-sample judge error compounds. The paper does not report how reliable the judge is on responses to perturbed prompts. In addition, one of the two examples in Table 8 reads as a hedged analysis rather than harmful content.
  • The hardening claims go beyond the evidence. The training recipe also includes affirmative-prefix augmentation, and the paper does not separate its effect from that of the SEEK data. There is no comparison with negatives from other methods, and over-refusal is not evaluated. The abstract's "from 90–100%" does not match the paper's own tables (PiF 67–75%, TAP/PAIR 21–30%).
Questions for authors

Same as weaknesses above.

3 comments

Limited novelty over BoN

SEEK and BoN differ in their primary objectives and, consequently, in how they explore the prompt space. BoN is primarily designed as a jailbreak attack that seeks to elicit a successful harmful response. Its prompt augmentations are deliberately constrained to preserve human intelligibility, making the perturbed prompts resemble plausible user queries. In contrast, SEEK is designed as a residual unsafe-behavior discovery framework for safety hardening. It aims to collect a diverse set of unsafe responses rather than merely establish whether a successful jailbreak exists. Accordingly, SEEK explores a broader perturbation distribution that need not preserve prompt readability, allowing it to explore a wider range of input variations. Thus, while BoN primarily targets attack success, SEEK explicitly targets behavioral coverage by retaining the unsafe responses encountered across perturbations.

This distinction is reflected in our coverage analysis. In particular, our experiments show that retaining all unsafe responses discovered by SEEK provides substantially broader behavioral coverage than retaining only the first unsafe response per prompt, with the gap exceeding 13 percentage points at a generation budget of 1024 (Appendix I). This highlights the importance of collecting multiple residual failures rather than treating discovery as a binary success criterion. More importantly, these failures provide useful supervision for downstream safety hardening: DPO training on SEEK-discovered responses substantially reduces vulnerability not only to SEEK itself but also to diverse attack families, including optimization-based (GCG), embedding-based (ASETF), transfer-based (PiF), and attacker-LLM-based (TAP and PAIR) attacks. This cross-attack improvement suggests that the discovered failures capture residual behaviors that are useful beyond the particular perturbations used to uncover them.

Regarding the theoretical contribution, we acknowledge that monotonicity and Jensen's inequality are general mathematical principles and are not unique to SEEK. Our contribution is to formulate residual unsafe-behavior discovery in terms of behavioral coverage and use this formulation to characterize the effect of perturbation resampling. Specifically, our analysis establishes that coverage and discovery probability increase with the sampling budget, and that drawing a fresh perturbation for each generation achieves higher expected coverage and discovery probability than repeatedly sampling from a fixed perturbation, under the stated assumptions. The analysis therefore provides a formal basis for the design choice of resampling perturbations, while the empirical results demonstrate its practical value for discovering diverse residual failures and using them for safety hardening.

The main metric is not validated for this setting

We appreciate the reviewer’s concern regarding the reliability of judge-based ASR evaluation, particularly at large generation budgets. We would like to clarify that SEEK's primary objective is negative-sample mining: it aims to collect residual unsafe responses that can serve as negative examples for subsequent safety training, rather than simply maximize prompt-level ASR.

The reviewer correctly observes that, when ASR is defined by whether at least one generated response is classified as unsafe, the probability of a false-positive prompt-level judgment can increase with the number of generations. However, for negative-sample mining, assuming a constant per-response false-positive rate (\epsilon), the expected number of false-positive responses among (N) generations is (N\epsilon), while the expected fraction of generated responses incorrectly classified as unsafe remains (\epsilon), independent of (N). Thus, increasing the generation budget does not inherently increase the fractional false-positive error across the generated responses, although the purity of the selected negative set also depends on the prevalence of genuinely unsafe responses.

To further examine the sensitivity of our evaluation to judge-model choice, we repeated the ASR experiments using an alternative judge model. The resulting ASR values differ from those obtained with our original judge by only approximately ±5 percentage points across the evaluated settings. We also repeated the evaluation after denoising generated responses and observed only a marginal change in ASR. These results suggest that the overall discovery trends are relatively robust to the choice of judge and to noisy or partially incoherent responses.

The hardening claims go beyond the evidence

We agree with the reviewer’s concern and plan to address those issues in future.

The abstract's "from 90–100%" does not match the paper's own tables (PiF 67–75%, TAP/PAIR 21–30%)

Though the ASR for PiF, PAT, and PAIR are much smaller than 90 - 100 %, the ASR for the other methods are more than 90%. We will clarify this point further in future.

Reviewer_S6m1OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SEEK, a stochastic exploration method to generate unsafe responses by adding random noise to the input prompts. The main idea is that random perturbations can help the model explore low-probability regions and generate unsafe responses that are usually suppressed by safety alignment. The paper further uses the generated unsafe responses for DPO training to improve the model's safety against different jailbreaking methods.

In the evaluation, the authors compare SEEK with several existing jailbreaking methods on five safety-aligned LLMs and two benchmarks, showing that SEEK can achieve comparable or better attack success rates with lower computational costs. The paper also evaluates the behavioral coverage of the generated unsafe responses and shows that DPO training can improve robustness against different types of jailbreak attacks. It also discusses the effects of different perturbation strategies, sampling budgets, and perturbation intensity on unsafe response generation.

Critical strengths
  • The paper is well-written and easy to follow, with clear motivation and problem formulation.
  • The proposed method is simple and effective, with a clear explanation of the underlying idea.
  • The paper provides a clear formulation of the problem and a comprehensive evaluation of the proposed method.
Critical weaknesses
  • Unclear whether adding random noise is an efficient way to discover unsafe responses.
  • Limited novelty of the safety hardening results, as the cross-attack generalization may simply come from additional DPO training.
  • Unreliable unsafe response evaluation due to the use of a single small judge model.
  • Limited validity of the behavioral coverage metric, as it only measures coverage relative to GCG-discovered unsafe clusters.
Questions for authors

I think this paper proposes a nice idea with quite a few formulations and experiments. The paper is well-written and easy to follow. But I have some concerns and questions.

(1) It seems that the unsafe response generation method is mostly about adding noise to the input. I am not sure if this is the best way to generate unsafe responses. For example, if we know that harmful answers are in low-probability regions, it might be more efficient to directly explore these regions or generate unsafe responses with guidance from a validator [1][2]. I am convinced that a well-designed noise algorithm can help induce more unsafe responses, but I am not sure if the current method is the most efficient one. I would like to see more discussion about this, especially in comparison with other ways of exploring low-probability regions.

(2) The paper claims that the generated unsafe responses are diverse and can help improve the model's safety by feeding them back to the model, which helps the model learn to avoid unsafe responses, even under other jailbreaking methods. However, I suspect this is a well-known effect. As the authors mentioned, unsafe responses are usually in low-probability regions, and further DPO training on the generated unsafe responses can help suppress their probabilities, which could naturally improve the model's robustness against other jailbreaking methods. I don't see the novelty of this claim. In particular, I am not sure whether this improvement comes from SEEK's ability to discover diverse unsafe responses or simply from additional safety training. I would like to see more discussion about this.

(3) This paper only uses a small judge model to evaluate all the unsafe responses. I don't think this is a sufficiently reliable evaluation method. A judge model can be fooled by unsafe responses or even noisy responses, and I suspect that some responses may not be truly unsafe but are judged as unsafe by the judge model. I would suggest using a more robust evaluation method, such as multiple judge models [3] or human evaluation, to ensure the accuracy of the unsafe response evaluation.

(4) The behavioral coverage metric is defined relative to GCG-discovered unsafe clusters, rather than an independently established taxonomy of unsafe behaviors. Consequently, the reported coverage primarily measures SEEK's ability to recover GCG-associated response clusters, but does not establish its coverage of the broader unsafe-behavior space. In particular, it is unclear whether SEEK discovers novel failure modes missed by GCG. I would encourage the authors to report SEEK-only, GCG-only, and overlapping unsafe clusters, together with semantic validation of whether these clusters correspond to genuinely distinct failure behaviors.

References

[1] Huang, Yangsibo, et al. "Catastrophic jailbreak of open-source llms via exploiting generation." ICML 2024.

[2] Zhang, Zhuo, et al. "On large language models’ resilience to coercive interrogation." 2024 IEEE Symposium on Security and Privacy (SP)

[3] Qi, Xiangyu, et al. "Fine-tuning aligned language models compromises safety, even when users do not intend to!." ICLR 2024.

2 comments

We thank the reviewer for the insightful and constructive feedback. We like to clarify the confusion as follows:

Noise versus other ways of exploring low-probability regions

We agree that injecting noise is not the only way to explore low-probability regions, and we do not claim that SEEK is universally the most efficient exploration strategy. However, the cited methods address a somewhat different objective. Generation Exploitation [1] varies decoding configurations (e.g., temperature, top-p/kp/k, and system prompts) to elicit a harmful completion, while LINT [2] intervenes in autoregressive generation by forcing lower-ranked tokens at selected positions, guided by an intent classifier, and requires access to top-kk token predictions. Both methods primarily aim to elicit a harmful completion for a given prompt.

SEEK instead focuses on discovering a diverse collection of residual unsafe behaviors for subsequent safety training. Importantly, the behaviors discovered by SEEK are not merely artifacts of its particular perturbation strategy: our coverage analysis shows that SEEK recovers 83.6% and 85.8% of GCG-discovered unsafe behavior clusters on HarmBench and AdvBench, respectively, at a generation budget of 1024 responses. This coverage increases with the generation budget.

We acknowledge that validator-guided approaches may be more sample-efficient for eliciting particular harmful responses. However, approaches such as LINT, which guide exploration toward specific harmful continuations, may recover a relatively narrow set of unsafe behaviours rather than a diverse range of failure modes. In contrast, SEEK uses simple, repeated stochastic exploration and post-hoc response classification to discover diverse failures. Thus, we view SEEK as a complementary approach that trades targeted guidance for a simple inference-only exploration strategy.

Not enough novelty

We agree with the reviewer that suppressing unsafe responses through DPO is an expected effect of preference optimization, and we do not claim this effect itself as a novel contribution. Our contribution is instead to investigate whether stochastic exploration can efficiently discover a diverse collection of residual unsafe behaviors that provides useful supervision for safety hardening.

The key question is not simply whether DPO on unsafe responses improves safety, but whether the source and coverage of these responses matter for the resulting robustness. Our experiments provide initial evidence in this direction. First, DPO training on SEEK-discovered failures substantially reduces ASR across several attack families, including optimization-based (GCG), embedding-based (ASETF), transfer-based (PiF), and attacker-LLM-based (TAP and PAIR) attacks, which differ from SEEK's input-perturbation strategy. Second, our ablation in Appendix G shows that retaining up to 20 positive and negative responses per prompt, rather than only one, generally leads to lower post-tuning ASR across the evaluated attacks. Importantly, the two settings use almost the same number of DPO training steps, and hence expose the model to approximately the same number of training-example presentations. The primary difference is that one setting exposes the model to multiple response pairs per prompt, while the other uses only a single pair. This provides evidence that collecting multiple responses per prompt, and thereby covering a broader range of residual unsafe behaviors, contributes to more effective safety hardening. We acknowledge that further controlled experiments would help isolate the contribution of behavioral diversity more precisely.

Judge reliability

We agree with the reviewer that relying on a single judge model may introduce evaluation bias, particularly when assessing potentially unsafe responses. To examine the robustness of our evaluation, we conducted additional ASR experiments using a different judge model. The resulting ASR values differ from those obtained with our original judge by only approximately ±5 percentage points across the evaluated experiments. This suggests that our main findings are not highly sensitive to the choice of judge model.

Furthermore, to investigate whether noisy or irrelevant content in the generated responses affects the evaluation, we conducted additional experiments after denoising the responses. We observed only marginal changes in the resulting ASR values, suggesting that noisy response content has a limited impact on our evaluation results.

Coverage defined relative to GCG

We appreciate the reviewer's observation. We would like to clarify that our coverage metric is defined at the prompt level, measuring how well SEEK recovers the unsafe failure behaviors discovered by another method for a given prompt. Establishing an independent, exhaustive taxonomy of unsafe behaviors is challenging because the failure behaviors are prompt-dependent and there is no readily available ground-truth set of all possible unsafe responses for each prompt. We therefore measure coverage relative to the failure sets discovered by existing methods, choosing GCG as our primary reference because it is a strong white-box attack baseline.

Importantly, this trend is not specific to GCG. We have also conducted coverage experiments using BoN as the reference method and observed a similar trend: at a generation budget of 1024, SEEK recovers approximately 80% of BoN-discovered unsafe behavior clusters on both HarmBench and AdvBench, with coverage increasing as the response-generation budget grows. We will include these additional results in the revised manuscript to demonstrate that the observed coverage trend is consistent across different reference methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.