acceptodds
Under review as a conference paper at ICLR 2027

Beyond Mechanisms: An Event-Level Characterization of LLM Jailbreaks

Abstract

Recent work has uncovered multiple internal mechanisms underlying jailbreaks, including changes in harmfulness recognition, refusal, and so on. Mechanism-level perspectives, however, are inherently fragmented: different studies typically isolate different factors or processes involved in jailbreaks, often at different levels of abstraction. As a result, they provide important local explanations of jailbreak behavior, but do not by themselves characterize how these factors jointly change when a jailbreak succeeds. We instead shift the unit of analysis from mechanism to event, asking which underlying factors, taken together, characterize the successful jailbreak event. Because reduced refusal does not necessarily imply substantive assistance toward the harmful goal, we treat harmful compliance as a separate outcome factor. To this end, we first investigate how harmfulness recognition, refusal, and harmful compliance jointly change during a successful jailbreak event, and find a consistent pattern across three LLM families: harmfulness recognition remains largely unchanged, while refusal decreases and harmful compliance increases. This event-level finding naturally motivates us to formulate jailbreak suppression as a constrained problem: increasing refusal and reducing harmful compliance under the constraint that harmfulness recognition remains approximately unchanged. We then implement this formulation with a constrained restoration method. Experiments across three LLM families show that the resulting method consistently reduces jailbreak success.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Causal representation learning, AI safety, Jailbreak, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SCmGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an event-level perspective on LLM jailbreaks, characterizing successful jailbreaks through three safety-relevant factors, harmfulness recognition (H), refusal (R), and harmful compliance (C). By analyzing the transition from direct refusal to successful jailbreaks for the same harmful goal, the authors observe that harmfulness recognition remains approximately unchanged, while refusal decreases and harmful compliance increases. Based on this observation, the authors formulate jailbreak suppression as reversing the changes in R and C while preserving H.

Critical strengths
  1. The paper introduces a meaningful event-level perspective on jailbreak analysis. Rather than examining individual safety-related mechanisms in isolation, it investigates how multiple factors jointly change during successful jailbreaks. This perspective is well motivated and offers a potentially novel framework for systematically characterizing jailbreak behavior beyond individual mechanisms.
Critical weaknesses
  1. Given that DHR and JS samples are selected based on refusal and harmful-compliance outcomes, respectively, and that R/C probes are trained using corresponding response-level labels, observing decreased R and increased C in the DHR–JS transition is largely expected. Sec. 2.2 shows that R and C capture different information, but they may still be strongly negatively correlated. It therefore remains unclear what additional insights their joint analysis provides beyond these expected relationships, limiting the novelty of the event-level characterization.

  2. The analysis identifies approximate H-preservation as an existing property of the DHR–JS transition, yet the proposed method makes explicit H-preservation a central design choice. The necessity of this design is unclear. As reported in Appendix E.3, the unprojected steering direction already satisfies H-equivalence across models, while Table 1 shows that the additional H-preservation yields only marginal ASR improvements (0.75–1.16 percentage points). Moreover, the comparison with refusal-direction steering uses different gating and intervention settings, making it difficult to establish whether the proposed method offers advantages over simpler activation steering approaches under comparable conditions.

Questions for authors
  1. Given the inherent relationship between refusal and harmful compliance, together with the outcome-based selection of DHR/JS samples and the supervision used to train R/C probes, how should the observed decrease in R and increase in C be interpreted beyond the expected differences between these two groups? What additional insights does the joint event-level characterization provide?

  2. Given that the unprojected direction already satisfies H-equivalence, can the authors clarify what additional benefit the component-level protection provides? In addition, can the effects of the steering direction, H-protection, and gating be better disentangled, particularly when comparing against steering-based baselines?

Reviewer_68F2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper probs the topic of LLM jailbreaking through a multi-factor analysis lens by considering jailbreak success as an event combining three distinct components: Harmfulness recognition (R), Refusal (R), and harmful compliance (C). The authors find a pattern in which R remains unchanged while R decreases and C increases. The insight that successful jailbreaks do not incur a substantial loss in harmfulness recognition motivates the construction of a jailbreak suppression method that applies the reverse change between successful jailbreak and direct refusal by increasing R and decreasing C. Results across three LLMs show a decrease in ASR with minimal loss on benign input utility.

Critical strengths
  • The decoupling of a successful jailbreak into the three components of harmfulness recognition (H), refusal (R), and harmful compliance (C) is novel and interesting to the LLM safety and jailbreaking community. This is evident as existing works largely fail to untangle all these factors, and more specifically refusal and harmful compliance, which are often considered antonymous.

  • The paper is overall well-structured and presents the main problem with a strong background in jailbreak literature.

  • The experimental pipeline is well defined, in which the authors use three diverse instruction-tuned LLMs. The choice to fix both attack families and goal is sound, to compute ASR to a set of unseen prompts on a fixed goal. Additionally, the justification for the use of a subspace to encode H is sound (Section 4.2 and Figure 2).

  • The prosed jailbreak suppression method shows big improvement in the ASR drop, especially with Llama-3.1-8B dropping from 22.16% to 2.48% using component-level protection.

Critical weaknesses

Major weaknesses

  • The setup of the paper sets harmful compliance as a metric for the model aiding toward the harmful goal, however just before that in lines 136-137, the question the authors attempt to answer is whether the model provides harmful assistance toward the requested goal (i.e., harmful compliance) despite containing refusal. These two are not the same, and the authors should clearly either change the question they claim to address or redefine the metric. Regardless, the question posed is a subset of the set containing all non-refused responses.

  • The authors repeatedly claim that state of the art works on jailbreak mechanism-analysis do not consider multiple factors when considering what constitutes a successful jailbreak. This is not entirely true, existing work does consider multiple factors [1,2], as do works that focus on jailbreak benchmark and evaluation [3], albeit not in the same manner suggested as "event-level" with the separate contribution of the three factors identified. This claim should be toned down. This is particularly important since the authors do specify this in other parts of the paper such as section 2.2 (lines 154-156).

  • In section 2.2 and 2.3, there are two main inconsistencies and drawbacks that take away from the empirical support of the main event-level characterization:

    • The authors justify their hypothesis concerning the separability of the harmful compliance and refusal both at the response and representation level. They find that both are distinct in the first through samples that contain both refusal and harmful assistance (between 23.1% and 44.6% of samples), and in the second through the non-collinearity of the refusal and harmful compliance linear probs. However, for the latter, the writing of the method is dense for someone that is not familiar with the literature, and additionally, the complexity is inflated as the use of the SVD can be simply reduced to computing the cosine similarity between the unit vectors of the refusal and harmful compliance.

    • Furthermore, the event-level analysis is defined as Δq(g)\Delta q(g) (2), which characterizes the change associated with a successful jailbreak event for a given harmful goal g defined as the difference between standardized scores for logits rq,l(x)r_{q,l}(x) for prompt x at layer l and for q∈{H,R,C}q \in \{H,R,C\} using their respective linear probs. Although this is a good metric, the authors go on to use Figure 3, which only plots the standardized scores. This in turn makes reading the figure difficult and the difference Δq(g)\Delta q(g) is not highlighted in the plots. The authors should plot the defined metric Δq(g)\Delta q(g).

[1] Zhao, Jiachen, et al. "Llms encode harmfulness and refusal separately." Advances in Neural Information Processing Systems 38 (2026): 140283-140318.

[2] Wu, Jinman, et al. "Knowing without acting: The disentangled geometry of safety mechanisms in large language models." arXiv preprint arXiv:2603.05773 (2026).

[3] Souly, Alexandra, et al. "A strongreject for empty jailbreaks." Advances in Neural Information Processing Systems 37 (2024): 125416-125440.

Minor weaknesses

  • Lack of citations to motivate the gaps in research: "harmfulness recognition, refusal, or other internal mechanisms" (line 145).

  • The token position used for training the probes is specified (in the supplementary material) to be the last non-padding prompt token. However, further clarification is needed on whether the prompt includes the chat template. This is crucial given the distinction of recent work on different positions where harmfulness and refusal are encoded [1]. Ideally, this should also be moved to the main text.

  • The reliance on Concept Component Analysis to identify the harmful subspace Q_H is unclear, and should be justified, ideally with an ablation study, or proper distinction. Otherwise, it seems arbitrary and may inflate complexity when a simple Spare Autoencoder suffices.

  • Some sections of the paper are not easy to read due to the relegation of important components to the supplementary material:

    • Many of the main results in Table 1 for the ASR reduction are not clear. The authors should include the initial baseline of ASR prior to post-suppression performance. It's one additional column and it would make the results easier to read.
    • The definition of Linear-H should be moved from page 24 to the main text.
    • The notation is inconsistent between the main text and the supplementary.
Questions for authors

The criteria for the choice of the attack families span token-level and prompt-level attacks. Does the event-level analysis and the suppression results hold consistently across families? Reporting per-attack family results could unveil interesting patterns.

Reviewer_zCBHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work argues that prior work largely focuses on individual mechanisms for understanding jailbreaks. It proposes a shift towards event-level analysis, in order to characterize the impact of multiple related factors. The work identifies three factors for their analysis: (1) harmfulness recognition, (2) refusal, and (3) harmfulness compliance. Empirically, the work finds that jailbreak events have a consistent pattern: harmfulness recognition is largely unchanged, refusal decreases, and harmfulness compliance increases for successful jailbreaks. The insight is used to develop a method for mitigating jailbreaks by activation steering. Experiments are conducted to show that jailbreak success rates can be reduced, while preserving the harmfulness recognition.

Critical strengths

Motivation is strong. Specifically, I agree that: Developing a holistic mechanistic understanding of jailbreaks is well-warranted; Most prior works have focused on individual components of jailbreaks; Methods that try to uncover a more complete mechanistic understanding should be more robust for jailbreak detection & mitigation, and should generalize better to newer model architectures & post-training recipes.

Critical weaknesses

(W1) Unclear novelty of the “event-level perspective”.

The motivation to study jailbreaks at the event-level is coherent, but I am unsure if it’s entirely novel (and thus, if it is appropriate for this work to claim it as a contribution). There has been some prior work on understanding how one can mechanistically move from a refused to successful jailbreak (or vice versa). See [1] and [2]. Though neither of these works formalizes “event-level jailbreaks” explicitly, they are interested in a transition between a refused request and a successful jailbreak (or vice versa). It is true that neither is specifically analyzing the H, R, C components identified in this work.

[1] Shubham Kumar and Narendra Ahuja, “Minimal, Local, Causal Explanations for Jailbreak Success in Large Language Models”, COLM 2026.

[2] Nathalie Maria Kirch et al. “What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks”, BlackboxNLP 2025.

(W2) Insufficient evidence that refusal and harmful compliance are actually different. There are two reasons why I say so.

Clarity: The distinction between refusal & harmful compliance is central to understanding the paper, and the abstract discussions on it did not clarify the difference. I strongly encourage the authors to put a qualitative example of the difference between the two (e.g., some example model response) in the main paper. The authors may consider replacing the current Fig. 1 with this (since I don’t feel the current Fig. 1 contributes much).

Imprecise definition: The paper claims that refusal and harmful compliance can co-occur. The truth of this critically depends on how the authors define refusal & harmful compliance. From the Appendix (lines 863-864), refusal is defined as “explicit refusal, policy invocation, or safety framing”, and harmful-compliance is defined as “substantive unsafe assistance”. I am most concerned about the refusal definition, which includes “safety framing”. Arguably, one could restrict the definition of refusal to only “explicit refusal”, which makes refusal & harmful-compliance mutually dependent (namely, refusal is the opposite of harmful-compliance).

I think the imprecise definition leads to cases where the paper may just be semantically mislabeling. More transparently disclosing the definitions and annotation procedure seems to be an important step. A human validation could also be important. Qualitative examples would be very helpful in illustrating the difference between the two (clearly, the authors see a difference, but I am failing to).

(W3) Experimental section can be strengthened with baselines.

Assuming, I am wrong about W2, and there is a difference between refusal & harmful compliance, the takeaways from the experiments still seem to be quite limited. A few reasons why I say that:

  1. The reduction in ASR from Component-H seems marginal (Table 1)
  2. Questionable if harmfulness is actually being preserved. The equivalence region defined in the paper seems to be a heuristic. One could argue that Gemma-2-9B is not actually preserving harmfulness.
  3. Insufficient comparisons. Comparing to [1] by itself is insufficient. I suggest the authors find, cite, and compare to stronger steering baselines (e.g., [2] and [3]) which try to also limit over-refusals. If the goal is not to achieve SOTA results, analyzing if AlphaSteer and AdaSteer perform because they implicitly do harmfulness preservation may also be an interesting mechanistic insight.
  4. Does including C improve jailbreak suppression? An experiment along this line could go a long way in empirically answering (W2).

[1] Andy Arditi et al. “Refusal in Language Models Is Mediated by a Single Direction”, NeurIPS 2024

[2] Leheng Sheng et al. “AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint”, ICLR 2026.

[3] Weixiang Zhao et al. “AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defender”, EMNLP 2025.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.