acceptodds
Under review as a conference paper at ICLR 2027

LLM Jailbreaks Exploit Attention Sinks

Abstract

Suffix-based jailbreak attacks append adversarial token sequences to harmful requests, bypassing safety guardrails in language models. Despite their effectiveness, the mechanisms enabling these attacks remain poorly understood. We find that tokens in adversarial suffixes are prone to inducing *attention sinks*—a phenomenon where certain tokens receive disproportionately high attention from subsequent tokens—and show that suffix-induced sinks drive attack success: amplifying the influence of suffix sinks improves attack success by up to 176%, while attenuating it reduces attack success by up to 84%. We trace this effect to the model's *refusal direction*: sink tokens induce perturbations that suppress the residual stream's refusal alignment across layers. Our results hold across four models and two suffix-based jailbreak methods, identifying attention sinks as a structural mechanism that adversarial suffixes exploit to bypass safety alignment.

Primary area:
interpretability and explainable AI
Keywords:
jailbreaking, adversarial suffixes, attention sinks, LLM safety, safety alignment, mechanistic interpretability, refusal direction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NNgpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper discusses how suffix-based jailbreaks are successful due to how adversarial tokens act as attention sinks. The authors make the paper easy to understand and is very concise with its findings.

Critical strengths
  • Most evidence presented in the paper seems to be backed up very well, with good sink subspace modulations too. It's also nice to see comparisons with AttnGCG in Appendix I.
Critical weaknesses
  • I feel like there is not a lot of novelty introduced by this paper. Several papers in the past [1, 2] talked about how these suffixes hijacked attention and how GCG suffixes especially do this. I think several of the claims could have been deduced from the above two papers.

[1] Arditi et al. Refusal in Language Models Is Mediated by a Single Direction. arXiv:2406.11717.

[2] Ben-Tov et al. Universal Jailbreak Suffixes Are Strong Attention Hijackers. arXiv:2506.12880.

Questions for authors
  • Out of curiosity, what happens when you amplify sinks in the fluent, permuted and random suffixes?
  • Does this hold for a fluent suffix attack, say in AutoDAN [3] or PAIR [4]?

[3] Liu et al. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. arXiv:2310.04451.

[4] Chao et al. Jailbreaking Black Box Large Language Models in Twenty Queries. arXiv:2310.08419.

6 comments
AuthorsOpenReview ↗

Response to Reviewer NNgp

We thank the reviewer for their review, and for their positive assessment of our evidence and of the AttnGCG comparison (now App. K). Figure and appendix numbers refer to the revision.

W1: Could the claims be deduced from [1, 2]?

We thank the reviewer for raising this. We agree that [1] and [2] are the closest prior work, and we have since added Section 6, which compares our work directly with them.

[2] show that suffixes hijack attention: in successful jailbreaks, attention from the chat tokens shifts toward the suffix. [1] observe a similar shift at the last chat token for one suffix on one model, alongside suppression of the refusal direction (their Figs. 5–6).

Yet even with both observations in hand, neither establishes how attention to the suffix leads to refusal suppression, and both name this as open: [1]'s suffix analysis "does not provide a comprehensive mechanistic understanding of the phenomenon", and [2] leave "the specific directions amplified by this mechanism" to future work.

Making this connection required three elements that neither paper provides:

  1. Individual sink tokens. Both [1] and [2] measure attention to the suffix as a whole, and [2] also pool over heads (Fig. 8). Such pooled scores cannot show which suffix tokens carry the effect. Measuring attention to individual suffix tokens in each head shows that it concentrates on a few sinks, which broadcast a shared low-rank perturbation to the tokens that follow (Fig. 2). This is what lets suffix attention act on a single direction.
  2. A targeted intervention. Amplifying only these sinks raises ASR by up to 125% while lowering refusal alignment across layers (Section 5), linking suffix sink attention to refusal.
  3. The response queries. Prior attention scores read the suffix's effect at the chat tokens. We find that response tokens keep attending to suffix sinks (App. M), and that including them in the intervention raises the mean ASR reduction from 54% to 79%, larger for every model and attack (Table 3).

These elements build on [1] and [2], but several of our findings would not be predicted from them: the effect concentrates on a few tokens per head, extends to the response, and arises only for optimized suffixes (Q1 below).

Q1: Amplifying sinks in the control suffixes

We thank the reviewer for this question. Prompted by it, we ran this experiment on Llama-3 and added it as App. H (Fig. 12).

Amplification raises ASR substantially only for the adversarial suffixes: GCG rises from 21.8% to 47.4% and PGD from 20.3% to 34.4%, whereas the fluent and random suffixes stay within 1 point of their baselines and the permuted suffixes gain at most 3.8 points. Sinks alone therefore do not jailbreak the model; the effect depends on what the sinks of an optimized suffix broadcast.

Q2: Fluent attacks (AutoDAN, PAIR)

We thank the reviewer for suggesting this extension. In response, we devoted substantial effort to running AutoDAN [3] on Llama-3, generating about 100 suffixes and evaluating them on JailbreakBench under the same protocol as GCG and PGD.

However, AutoDAN is weak in this setting: its top-10 suffixes reach an ASR of only 5.2% (best: 7%), at the level of our non-adversarial fluent control (5.9%) and well below GCG (21.8%). This is consistent with prior reports: [5] (Table 1) find that AutoDAN reaches 10% ASR on an AdvBench subset and 2% on MaliciousInstruct against Llama-3-8B.

With only about five successful prompts per suffix, shifts of a few points in either direction fall within noise, so this setting cannot reliably show whether modulating sinks affects fluent jailbreaks.

PAIR [4], in turn, rewrites the entire prompt rather than appending a suffix, so it has no suffix positions to modulate; it lies outside the suffix-based setting we study.

We thank the reviewer again for their feedback, which has helped us improve the paper.

[1] Arditi et al., 2024; [2] Ben-Tov et al., 2025; [3] AutoDAN (Liu et al., 2024); [4] PAIR (Chao et al., 2023); [5] Wu et al., Monte Carlo Tree Search Based Prompt Autogeneration for Jailbreak Attacks against LLMs, COLING 2025.

AuthorsOpenReview ↗

Follow-up on Q2: Does this hold for a fluent suffix attack such as AutoDAN?

We wanted to follow up on the reviewer's Q2. In our initial response, we found AutoDAN [3] too weak on Llama-3 (top-10 ASR of 5.2%) to test whether sink modulation affects it. We have since run AutoDAN on Gemma-2, where it is much stronger: its top-10 suffixes reach an average ASR of 40.1% on JailbreakBench. We applied the same sink-subspace modulation as for GCG and PGD, at the Gemma-2 threshold of Fig. 5 (ϵ=0.0125\epsilon = 0.0125), and added the results as App. P (Fig. 20). The results show two things:

  • Sink modulation moves AutoDAN in the same direction as GCG and PGD. ASR rises steadily with λ\lambda up to λ=2\lambda = 2: ablating the suffix sinks (λ=0\lambda = 0) lowers it from 40.1% to 27.9%, a 30% relative reduction, and amplifying them raises it to 44.1% at λ=2\lambda = 2.
  • The effect is smaller than for GCG and PGD. On Gemma-2 at the same threshold, ablation lowers GCG from 55.2% to 4.4% and PGD from 71.0% to 7.6%. Due to compute constraints, we selected the AutoDAN suffixes from 80 candidates rather than the 576 we use for GCG and PGD. In our experience, modulation is most effective on the most successful suffixes, so a larger pool may yield a stronger effect.

Suffix sinks therefore also contribute to the success of a fluent suffix attack, not only of token-level optimized ones.

Should the reviewer feel this addresses their concern, we would be grateful for any reconsideration of the score.

[3] AutoDAN (Liu et al., 2024).

AuthorsOpenReview ↗

Thank you and a note before the discussion closes

As the discussion period draws to a close, we thank the reviewer once again for their thoughtful review, which has helped us improve the paper. In response, we added a direct comparison with [1, 2] (Section 6). Prompted by their questions, we also ran sink modulation experiments on our fluent, permuted, and random control suffixes (App. H) as well as on AutoDAN suffixes (App. P), a fluent suffix attack. We would be grateful for a brief acknowledgment of our response and, if the reviewer feels it addresses their concerns, for reconsideration of the score.

Reviewer_N5CkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper performs a mechanistic analysis of suffix-based jailbreaks (focusing on GCG and PGD). It shows that such suffixes introduce "attention sinks": suffix tokens that attract a disproportionate amount of attention from subsequent tokens (i.e., chat-template tokens and the generated response). Through causal experiments, the paper shows it is possible to amplify and suppress the jailbreak effect. Further, the paper connects the sinks' contribution in internal representations to the so-called refusal direction, introduced in prior work.

Critical strengths
  • The paper is clearly written and organized, making it easy to follow.

  • The paper convincingly shows, through various experiments, both the emergence and significance of attention sinks in suffix-based jailbreaks.

  • By this, the paper contributes to the mechanistic understanding of suffix-based jailbreaks.

Critical weaknesses
  • Unclear novelty. Prior work--as also stated by the paper--has shown in different ways that adversarial suffixes (e.g., GCG) introduce internal representations that often attract attention from subsequent tokens (e.g., from chat-template tokens), and suppress certain harmfulness directions. It is unclear how the paper is positioned wrt this line of work, and it seems that the paper mainly serves as another piece of evidence for this known phenomenon. It would be helpful if the paper were more concrete about its position wrt this prior work, how it further extends/challenges this existing line of work, and generally to clarify its novelty.

  • Lacking in-depth exploration of sinks. The attention-sink score offers a coarse measure for attention sinks; in particular, averaging over all subsequent token positions may obscure which specific query positions (k) are affected the most by the sink token. It would be helpful to complement the current analysis to show which token positions are more susceptible to these sinks: are these usually the chat-template tokens? The beginning of the response? Or perhaps it is when the harmful response gets to concrete harmful information?

  • Lacking discussion (e.g., of future work). It would be interesting to discuss in the paper what sorts of applications the insights of the analysis might provide. In particular, how may future work apply the results of the paper's analysis to strengthen red-teaming techniques, or to build defenses?

Questions for authors

Could you address the weaknesses listed above?

4 comments
AuthorsOpenReview ↗

Response to Reviewer N5Ck

We thank the reviewer for their detailed review, and for finding the paper clear and the experiments convincing. Figure and appendix numbers refer to the revision.

W1: Positioning relative to prior work

We thank the reviewer for this suggestion. We agree that prior work has shown that suffixes draw attention from subsequent tokens [1–3] and suppress the refusal direction [4]. To make our position concrete, we have since added Section 6, which compares our work directly with the closest prior work.

These works find that in successful jailbreaks, attention from the chat tokens following the suffix shifts from the instruction toward the suffix (Fig. 8). Their scores, however, are coarse and fixed: each reads the same region for every input, pooled over heads and all suffix tokens, so they cannot show which suffix tokens carry the effect.

Our approach is more surgical: for each input, we identify the individual suffix tokens that act as sinks in each head and intervene on exactly those. Since each head has only a few sinks, if any, we modulate a small subset of suffix tokens per head and leave heads without sinks untouched. This extends and challenges prior work in two ways.

Extends: from attention to refusal. Prior work shows that suffixes attract attention and that refusal is suppressed, but not how one leads to the other, which [4] and [2] name as open: [4]'s suffix analysis "does not provide a comprehensive mechanistic understanding", and [2] leave "the specific directions amplified by this mechanism" to future work. This per-token, per-head view reveals a four-step mechanism linking the two (Fig. 2):

  1. suffix attention concentrates on a few sink tokens per head;
  2. sinks broadcast a shared low-rank perturbation to the chat and response tokens;
  3. this perturbation lowers alignment with the refusal direction;
  4. scaling it steers ASR in both directions.

Challenges: the suffix keeps acting on the response. Prior attention scores read the suffix's influence at the chat tokens.† We show that response tokens keep attending to suffix sinks (see W2), and that this matters: restricting our intervention to chat queries, the query scope of prior methods (Fig. 16), weakens it. Including response queries raises the mean ASR reduction from 54% to 79%, larger for every model and attack (Table 3).

† AttnGCG [1] is a partial exception: it rewards attention from a short, fixed target string during optimization, but does not analyze the model's own response, and its pooled score tracks ASR on only 2 of 5 models (App. K).

W2: Which query positions do sinks affect?

We thank the reviewer for this question, and agree that the sink score alone cannot answer it. Because αt\alpha_t is a mean over all subsequent queries, it does not constrain how attention is spread across them: a token can pass the threshold on attention from a handful of queries, such as the chat tokens, while the response ignores it. Prompted by this question, we have added App. M (Fig. 17), which records the attention each query position pays to each sink, over the chat tokens and the first 145 response tokens.

The chat tokens matter most: attention per sink peaks there, consistent with prior work. However, the sinks stay attended throughout the response. At every response position, a sink receives 9–32× the attention of a non-sink suffix token, and after the first few response tokens this ratio is roughly constant. The sinks' influence thus extends across the response rather than being confined to its start.

This attention also matters causally: adding response queries to the intervention raises the mean ASR reduction from 54% to 79% (Table 3, App. L).

W3: Applications to red-teaming and defenses

We agree this discussion strengthens the paper, and have since added an Implications paragraph to the Conclusion. Attention sinks are widespread in language models and have mostly been studied as benign or useful structure; we show they are also an attack surface. This suggests concrete directions:

  • Defenses. Attenuating suffix sinks lowers ASR by up to 99% by restoring coherent refusals (App. N). Our intervention uses the known suffix position; a natural next step is to detect adversarial sinks without this knowledge, for monitoring or inference-time intervention. Architectures that control sinks, such as gated attention [5] or registers [6], offer another lever.
  • Attacks and red-teaming. Amplifying sinks raises ASR by up to 125%, suggesting attack objectives that reward sink formation, and red-teaming evaluations could report how readily adversarial inputs form sinks, as a measurable indicator of vulnerability.

We thank the reviewer again for their feedback, which has helped us improve the paper.

[1] AttnGCG (Wang et al., 2024); [2] Ben-Tov et al., 2025; [3] Attention Slipping (Hu et al., 2025); [4] Arditi et al., 2024; [5] Qiu et al., 2025; [6] Darcet et al., 2024.

AuthorsOpenReview ↗

Thank you and a note before the discussion closes

As the discussion period draws to a close, we thank the reviewer once again for their thoughtful review, which has helped us improve the paper. In response, we added a section positioning our work directly against prior attention-based work (Section 6), an analysis of which query positions attend to suffix sinks (App. M), and a discussion of implications for defenses and red-teaming (Conclusion). We would be grateful for a brief acknowledgment of our response and, if the reviewer feels it addresses their concerns, for reconsideration of the score.

AuthorsOpenReview ↗

Final follow-up before discussion closes

As the discussion period closes tomorrow (Nov 18, AoE), we wanted to gently follow up. We would greatly appreciate the reviewer's feedback on our response and revisions, and we are happy to clarify any remaining questions in the time left.

AuthorsOpenReview ↗

A final note in the last hours of discussion

With only a few hours left in the discussion period, we wanted to reach out one last time. Another reviewer has indicated that our response addressed the majority of their concerns and has raised their score. For convenience, here is where each of the reviewer's points is addressed:

  • Positioning and novelty (W1): a direct comparison with prior attention-based work, showing how we extend and challenge it (Section 6).
  • Which query positions sinks affect (W2): per-position attention to sinks over the chat and response tokens (App. M), and the effect of including response queries in the intervention (Table 3).
  • Applications to red-teaming and defenses (W3): a new Implications paragraph (Conclusion).

We would sincerely value the reviewer's view on whether these address their concerns. Any brief acknowledgment would be greatly appreciated.

Reviewer_FQxfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how attention sinks contribute to suffix-based LLM jailbreaks. The authors show that adversarial suffixes induce more attention sinks than normal suffixes. By selectively amplifying or suppressing sink contributions, they can increase attack success rate by up to 176% or reduce it by up to 84%. They further suggest that attention sinks facilitate jailbreaks by suppressing the model's refusal direction.

Critical strengths
  • Interesting mechanistic insight. The paper connects attention sinks to jailbreak effectiveness, providing a new perspective on how adversarial suffixes bypass safety alignment.

  • Strong experimental evidence. The intervention experiments demonstrate consistent effects across four models and two attack methods, supporting the importance of attention sinks.

Critical weaknesses
  • Limited novelty and significance. Prior work has already demonstrated that adversarial suffixes manipulate attention patterns to facilitate jailbreaks. This paper further associates this phenomenon with attention sinks, but the conceptual advance appears relatively incremental. More importantly, the findings do not lead to a new attack method, effective defense, or substantially improved understanding of how safety alignment fails. The practical and broader scientific implications therefore remain limited.

  • Insufficient evidence for the claimed mechanism. Although the intervention experiments show that modifying sink contributions affects ASR, they do not establish that attention sinks are the fundamental mechanism behind suffix-based jailbreaks. Attention sinks may simply be a correlated feature of adversarial suffixes. Similarly, the observed reduction in refusal-direction alignment does not establish a causal relationship with jailbreak success. The absence of benign-task evaluations also makes it difficult to distinguish targeted safety effects from general disruption of model computation.

Questions for authors
  • Novelty and significance: How does the proposed attention-sink mechanism provide fundamentally new insights beyond existing work showing that jailbreak suffixes manipulate attention distributions?

  • Causal mechanism: Can the authors demonstrate that suppressing the refusal direction is necessary for the increased jailbreak success, rather than merely a correlated consequence of sink amplification?

  • Intervention specificity: Does sink attenuation preserve benign instruction-following performance? How can the authors rule out the possibility that reduced ASR results from general degradation of model behavior?

5 comments
AuthorsOpenReview ↗

Response to Reviewer FQxf

We thank the reviewer for their thoughtful review and for recognizing the mechanistic insight and the strength of our experimental evidence. Figure and appendix numbers refer to the revision.

W1 / Q1: Novelty beyond prior work on attention manipulation

We thank the reviewer for raising this. We agree that prior work [1–4] has established that suffixes draw attention toward themselves, and we now make our relation to it explicit in a new Section 6. These works typically consider attention from the chat tokens following the suffix, pooled over heads and all suffix tokens (Fig. 8). They show that suffixes attract attention, but not how this attention suppresses refusal, which two of them name as open: [4]'s suffix analysis "does not provide a comprehensive mechanistic understanding", and [2] leave "the specific directions amplified by this mechanism" to future work.

Measuring attention to individual suffix tokens in each head answers this with a four-step mechanism (Fig. 2): (1) suffix attention concentrates on a few sink tokens per head; (2) sinks broadcast a shared low-rank perturbation to the chat and response tokens; (3) this perturbation lowers alignment with the refusal direction; (4) scaling it steers ASR in both directions. A consequence that prompt-side scores miss is that the response matters: sinks receive 9–32× the attention of non-sink suffix tokens at every response position (App. M), and restricting our intervention to chat queries, the query scope of prior methods (Fig. 16), weakens it. Including response queries raises the mean ASR reduction from 54% to 79%, larger for every model and attack (Table 3).

W1: Practical and scientific implications

We agree these implications deserve explicit discussion, and have since added an Implications paragraph to the Conclusion. Our results pinpoint a safety alignment failure: a few suffix sinks broadcast a refusal-suppressing perturbation. This suggests concrete directions:

  • Defenses. Attenuating suffix sinks lowers ASR by up to 99% by restoring coherent refusals (App. N), making sinks a natural target for monitoring or inference-time intervention.
  • Attacks and red-teaming. Amplifying sinks raises ASR by up to 125%, suggesting attack objectives that reward sink formation, and red-teaming evaluations could report how readily adversarial inputs form sinks.

W2: Are sinks only a correlated feature?

We thank the reviewer for this important point. If sinks were merely correlated with attack success, modulating them would do no more than modulating other suffix positions. Our budget-matched position control shows otherwise (App. G, Fig. 11): modulating the same number of randomly chosen positions in the same suffix, per head, yields much smaller shifts. The sink intervention is larger on all five models under attenuation and four of five under amplification, by 2–11× and 2–7× (Llama-3: +67% vs. +10%).

W2 / Q2: Is the refusal link causal?

We agree that lower refusal alignment under amplification does not by itself show that this suppression drives success in real attacks. Our account is a two-step chain: suffix sinks → refusal suppression → compliance. We establish the first step by intervention: amplifying suffix sinks lowers refusal alignment across layers, graded in the sink gain λ\lambda (Section 5, App. I). The second is established by [4]: erasing the refusal direction "prevents it from refusing harmful instructions".

Prompted by the reviewer's question, we have added App. J (Fig. 15), testing whether this suppression separates success from failure without intervention. Using the refusal-alignment measurement of [4], we compare 50 successful and 50 failed attacks per model and attack. Failed attacks keep the model's usual refusal peak near the extraction layer; successful attacks show a much smaller peak or none. All differences are statistically significant (Holm-adjusted p<10−3p < 10^{-3}).

W2 / Q3: Is the ASR drop due to general degradation?

We agree that a lower ASR alone could reflect broken generation rather than restored safety. Prompted by this question, we have added App. N (Fig. 19), where an LLM judge (Claude Opus 5.5) labels each failed response as refusal, deflection, compliance or incoherent, comparing failures without intervention to those created by the strongest attenuation (ϵ=0.005\epsilon = 0.005, λ=0\lambda = 0) on Gemma-2 and Llama-3.1. None of the attenuation-created failures are incoherent and ≤ 0.6% are compliant, while explicit refusals rise for every model and attack, from 58–69% to 80–92%. Attenuation thus restores the model's standard refusal behavior rather than degrading its output.

We thank the reviewer again for their feedback, which has helped us improve the paper.

[1] AttnGCG (Wang et al., 2024); [2] Ben-Tov et al., 2025; [3] Attention Slipping (Hu et al., 2025); [4] Arditi et al., 2024.

AuthorsOpenReview ↗

Follow-up on W2 / Q3: Does sink attenuation preserve benign instruction following?

We wanted to follow up on the reviewer's Q3: does sink attenuation preserve benign instruction following? Our initial response showed that on harmful prompts, attenuation produces coherent refusals rather than broken output (App. N). To complete the picture, we have now evaluated attenuation on benign tasks directly, as the reviewer suggested (App. O, Table 7).

On Llama-3.1, we measure IFEval [5] accuracy on benign prompts with each of the top-3 GCG suffixes appended, with and without the strongest sink attenuation (ϵ=0.005\epsilon = 0.005, λ=0\lambda = 0):

ConditionIFEval accuracy (%)
No suffix78.7
Suffix50.9
Suffix + sink attenuation60.8

The table shows that attenuation does not degrade benign performance:

  • Attenuation raises accuracy. When the suffix is present, attenuating its sinks raises accuracy by 10 points, from 50.9% to 60.8%.
  • The drop below the no-suffix baseline comes from the suffix, not from attenuation. Appending a suffix alone lowers accuracy from 78.7% to 50.9%, mostly by making the model refuse benign requests: 84% of the prompts the suffix breaks fail with an explicit refusal. Attenuation removes many of these refusals: 81% of the prompts it recovers are ones where the suffix had caused a refusal.

Together with App. N, where attenuation turns jailbreaks into coherent refusals, this shows that attenuation removes the suffix's influence rather than disrupting the model in general.

Should the reviewer feel this addresses their concern, we would be grateful for any reconsideration of the score.

[5] IFEval (Zhou et al., 2023).

AuthorsOpenReview ↗

Thank you and a note before the discussion closes

As the discussion period draws to a close, we thank the reviewer once again for their thoughtful review, which has helped us improve the paper. In response, we added a direct comparison with prior attention-based work (Section 6), an analysis showing that refusal alignment separates successful from failed attacks (App. J), and two analyses showing that sink attenuation restores coherent refusals on harmful prompts (App. N) and preserves benign instruction following (App. O). We would be grateful for a brief acknowledgment of our response and, if the reviewer feels it addresses their concerns, for reconsideration of the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.