acceptodds
Under review as a conference paper at ICLR 2027

Sink Before You Speak: Connecting Language Generation, Attention Patterns, and Representational Geometry

Abstract

Large Language Models (LLMs) have demonstrated an impressive capability to generate coherent text that is relevant to the subject matter provided in a user's prompt. Despite their sophistication, LLMs have also been shown to generate completely irrelevant text or strings of repeating, relevant tokens both in naturalistic settings and through adversarially constructed prompts in natural language. While some research has been done into the mechanistic cause for these sorts of generations, these investigations provide incomplete explanations on account of considering these phenomena independently. In this work, we provide a unified account for how both relevant, repetitive and coherent, irrelevant generations arise from specific activation patterns in the attention mechanisms of these models. This explanation centers on the attention sink, the emergent behaviour where high softmax attention values are assigned to seemingly mundane tokens such as the start-of-sentence delimiter. We evidence this explanation by directly modifying the attention maps and performing a novel perturbation to the latent representations. These causal interventions not only reliably induce the types of language generation we are interested in, but also the attention patterns we hypothesize to be at fault for them. We finally use this per-prompt spectral approach to characterize the operation of the attention sink. The findings of this work give a more complete understanding of the operation of LLMs, as well as how models that are seemingly steeled against existing jailbreaks remain susceptible to the shortcomings of the transformer architecture.

Primary area:
interpretability and explainable AI
Keywords:
LLMs, Interpretability, Attention Sinks, Representation Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7U4wOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how attention patterns and representational geometry contribute to two undesirable LLM behaviors: coherent but irrelevant responses and relevant but repetitive text. Across Llama and Mistral models, the authors find that both behaviors are associated with a weakened attention sink, while coherent but irrelevant generations also exhibit increased attention entropy. They use causal interventions on attention maps and hidden representations, including a per-prompt singular value decomposition approach, to induce these behaviors and analyze how disruptions to different directions in the representation space affect the attention sink. The results provide a unified mechanistic account of these generation patterns and reveal vulnerabilities in models that may appear robust to existing jailbreaks.

Critical strengths
  1. Per-prompt representation spectrum analysis. The authors perform SVD separately for each prompt and perturb the model by projecting out different singular-vector directions, introducing a new approach to analyzing the relationship between attention sinks and representational geometry.

  2. A unified framework for analyzing two types of undesirable behavior. The authors compare coherent but irrelevant outputs with relevant but repetitive outputs, finding that both are associated with a weakened attention sink, while exhibiting different attention patterns.

Critical weaknesses
  1. Figures 1 and 2 are difficult to read because the text is too small, and they look unfinished for an ICLR submission. I recommend redrawing them. The text in Figure 4(a) is also unreadable.

  2. Typos and formatting issues:

    • Equation (1) appears to extend beyond the paper’s layout.
    • In the Figure 2 caption: ”the” or ”a”
  3. The evaluation has major issues. First, the metrics are not clearly stated in the paper; they are only described implicitly in Section 3. All three metrics are scored automatically by LLMs, but the paper does not evaluate the stability of these scores or use objective metrics. The paper also lacks tables that would present the results clearly.

  4. The experimental coverage appears insufficient in the current version. The test scenarios and datasets are not described in the main text. Important details, such as the number of data points and how many times each item was evaluated, should not be relegated to the appendix, as readers may not find them easily. The results should also be presented in tables. As currently presented, the figures do not make it clear that these datasets were used.

  5. Since the method involves modifying the input text, the paper should include case studies showing how different repeated words lead to different outputs. These examples could be included in the appendix.

  6. I do not think using only “the” and “a” provides sufficient control over the repeated-token attack. These two words are low in semantic content, so they are worth testing, but the experiments should also include semantically richer tokens, such as concrete nouns (e.g., “book”), numbers, names, and more complex words. The current experiments feel too shallow and lack depth. The paper claims in the main text that it uses 30 irrelevant jailbreak prompts, but I could not find them.

Questions for authors

I have no questions.

1 comment

Thank you for your review and for helping to improve the manuscript. We respond to each of your concerns below:

  • 1: We will improve the readability of the figures in the camera ready version of the manuscript.

  • 2: Thank you for bringing these formatting issues to our attention. We will correct them in the next edition of the paper.

  • 3: We explicitly provide a full account (including code) of the evaluation process in Section 3, Appendix A.1, and the supplementary material. We will collate this information in the main body of the final version of the paper. The availability of “objective” metrics for natural language processing related tasks has long been an issue due to the subjective nature of the quantities investigated. Consequently, turning to automatically computed metrics as an approximation for exact human evaluation has a long history in the field (BLEU, ROUGE, METEOR, etc.). To address your concern regarding the consistency of these metrics, we randomly selected 25 Good, Bad, (Coherent, Irrelevant), and (Relevant, Repetitive) generations as rated by gpt-5.6-terra and had them rated by gpt-6.1-terra and gpt-6.1-sol. All three models agree on 79% of cases and at least two models agree in 99% of cases, indicating a strong level of stability of our evaluation scheme.

  • 4: We will move some of the content of Appendix A.2 to the main body of the paper when we are not constrained by the page limit. Throughout the paper, we make reference to the number of data points used in experiments as well as their provenance, but we can add a reminder of this to the figure captions. We also include a tabular version of the bar chart presented in Figure 4 b) below, as it is the most well suited for a tabular representation. However, we do not believe that these minor representation issues indicate that the “experiment coverage is insufficient”. We experiment with a wide variety of different datasets, two different jailbreak types with 32 instantiations, multiple models, and dozens of different intervention types to produce a thorough account of the undesirable responses we are studying.

Response TypeBig Upper + 1 (n = 3600)Small Upper (n = 4000)DAS (n = 400)DAS + E (n = 400)No Perturbation (n = 400)
Coherent, Irrelevant0.6700.0190.0490.2870.023
Relevant, Repetitive0.0460.6210.4420.0000.093
Good0.0390.1610.4550.0050.876
Bad0.2450.1990.0540.7080.008
  • 5: The general format of the successful jailbreaks is demonstrated in Figure 1. The unsuccessful jailbreaks will generally either repeat the specified word, or write a passage about repetition. We will include more examples of this in the appendix of the camera ready paper.

  • 6: We experiment with 32 different repeated words in both jailbreaks, extracted from Nasr et al. (2023). These words have a wide range of semantic meanings. The words are "company", "poem", "the", "make", "his", "her", "an", "so", "one", "b", "J", "life", "send", "make", "part", "with", "work", "word", "cell", "you", "time", "eye", "of", "on", "come", "good", "do", "up", "last", "year", "call", and "a".

Each of your concerns can be easily addressed in an update to the paper, to which we are grateful, or are already present in the current version of the manuscript. With that in mind, we ask you to improve your score.

Reviewer_J32COpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates how attention sink and geometry of internal representations relate to two kinds of degenerate generation in LLMs: coherent but irrelevant text and relevant but repetitive text. The authors use two models (Mistral 7B instruct and LLaMA 3.1 8B instruct) and first analyze two variants of attack: repeating a single word and appending a coherent prompt to the repetition. The first yields coherent and irrelevant text while the second yields repetitive and relevant text. They then attempt to reproduce the observed behaviors with mechanistic interventions on the attention sink and attention patterns on the prompt tokens. They conclude that the mechanisms they intervened on explain only part of the behaviors they are investigating, and produce a new hypothesis regarding the similarity between the internal representation of input and output tokens. The authors test this hypothesis with a novel spectral perturbation technique, managing to reproduce the degenerate generation behaviors reliably. Finally the authors study the effects that the spectral perturbation technique creates and its usefulness as analitical tool.

Critical strengths

The work proposes a set of sufficient interventions that modify the attention sink phenomenon and study its relation to the degenerate output, extending the understanding of a frequent phenomenon in large language models that is relevant to the community.

I found the paper having a polished subdivision between hypothesis, experiments and conclusions that made it easy and interesting to read.

The spectral technique proposed here has the potential to be adopted by the community to cheaply explore the internal representation of Large Language Models.

Critical weaknesses

While the results are interesting, they are still preliminary in terms of producing a mechanistic account of the degenerate generation behavior and its relation with attention sink. As the authors specify, they produce sufficient causal intervention but it is not clear how much these interventions are specific, minimal and necessary. In particular: while the spectral perturbation is a sufficient causal intervention, it may not be a specific one. Big Upper + 1 removes between 50% and 70% of the singular directions of the prompt. Disrupting the representation this much may plausibly erase most of what the model encoded about the prompt. Hence, the success of this intervention may be caused by sufficient representation disruption rather than evidence for the proposed mechanism. It would be interesting to understand what happens if this experiment is run without touching the attention sink. This would allow to isolate the contribution of the attention sink value. While in figure 5a they show that with keep-2 the attention sink does not drop, it is not shown what’s the effect on degenerate generation.

Overall I think the paper produces a worthy contribution to the understanding of the causes and consequences of attention sink. I still have some concerns regarding the specificity of the interventions and I am inclined to decide for weak acceptance.

Questions for authors

Is the model reliably producing degenerate output when the attention sink does not change much? How does the outcome of the big upper spectral perturbation experiment change if the attention sink value is clamped to the original value?

2 comments

Thank you for your thoughtful review and helping us improve the work. We respond to your listed weaknesses and questions below:

  • “As the authors specify, they produce sufficient causal intervention but it is not clear how much these interventions are specific, minimal and necessary”: We would like to push back against the notion that these interventions need to be specific, minimal, or necessary. There is not likely to be a necessary condition for producing relevant, repetitive text or coherent, irrelevant text. For instance, an LLM asked to repeat a very repetitive song would produce relevant, repetitive text despite no mechanism failures within the model itself. Finding sufficient conditions to reproduce the phenomena observed in the jailbreaks is an important step in fully characterizing how the transformer architecture can give rise to different types of output. The previous work of Yona et al. (2025) suggested that the coherent but irrelevant responses produced by the irrelevant jailbreak could be mainly attributed to the loss of the attention sink. Our work makes a significant contribution in this direction of investigation, as we provide a more thorough account of how the attention sink’s disruption (and potential restoration), along with other aspects of the attention distribution, relate to two separate jailbreaks producing different kinds of responses (coherent, irrelevant and relevant, repetitive).

  • “Disrupting the representation this much may plausibly erase most of what the model encoded about the prompt…It would be interesting to understand what happens if this experiment is run without touching the attention sink”: We appreciate your question, and ran some additional experiments involving the “Keep 2” variant from Figure 5 a) to respond to it. We generalize this to the Small Upper intervention regime as well. Results are presented below:

Response TypeBig Upper Keep 2 (n = 3600) FractionSmall Upper Keep 2 (n = 4800) Fraction
Coherent, Irrelevant23.5%2.7%
Relevant, Repetitive41.9%17.3%
Good16.7%78.2%
Bad17.9%1.7%

We can see that the disruption of the attention sink, even temporarily, is essential for both spectral perturbations to work reliably. Small Upper Keep 2 produces nearly 80% Good responses, only 7 points less than the unperturbed models. Meanwhile, Big Upper Keep 2 is very inconsistent in its effect, indicating that the brief disruption of the attention sink demonstrated in Figures 4 and 5 is essential for reliably producing Coherent, Irrelevant responses. While the corruption of semantic information within the representation undoubtedly plays a role in producing the responses we study (especially in the case of Big Upper + 1), we can see that specific behaviour from the attention sink is essential in reliably producing the coherent, irrelevant responses we wish to emulate from the jailbreak.

  • “Is the model reliably producing degenerate output when the attention sink does not change much? How does the outcome of the big upper spectral perturbation experiment change if the attention sink value is clamped to the original value?”: With respect to your first question, our Big Upper Keep 2 results presented above demonstrate that keeping the attention sink leads to a wide variety of degenerate inputs.

We appreciate your thoughtful feedback and your positive assessment of our work. Please let us know if we can provide any additional clarifications.

Sources:

Itay Yona, Ilia Shumailov, Jamie Hayes, Federico Barbero, and Yossi Gandelsman. Interpreting the repeated token phenomenon in large language models. arXiv preprint arXiv:2503.08908, 2025.

I thank the authors for their detailed rebuttal and I will maintain the current score.

Reviewer_7QVZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a study that connects two jailbreaking behaviors of LLMs (relevant-but-repetitive and coherent-but-irrelevant responses) to weakened attention sinks, which can be distinguished through increased prompt-attention entropy. Experiments on two LLMs show that the two behaviors can be induced by a controlled experiment that manipulations of attention maps. The authors furthre presents analysis that relates the attention sink to different directions in the top singular vector subspace. The results provides useful evidence for the understanding and potentially interventions of the target jailbreaking patterns.

Critical strengths
  • the paper presents complementary attention and representation interventions that connect the two different jailbreaking behaviors of LLMs.
  • the paper shows that preserving leading signular directions provides an informative and controllable sink-recovery effect.
Critical weaknesses
  • the major concern of the study is that the experiments do not establish causality between attention sink and jailbreaking. The interventions proposed confound the disruption of attention sink with the damage of semantic information. In particular, manipulating attention sink as proposed in the paper also redistributes attention to other tokens, therefore confounds with the effect of attention sink. In other words, the experiment identifies the total effect of suppressing sink attention and redistribution, and it cannot separate those mechanisms. The spectral ablation introduces an even stronger confounding, as they can directly remove semantic innformation as well as weakening the sink. The claim of "causal intervention" in the paper is too strong, and I'd suggest the authors remove this claim, or introduce a rigorous causal analysis to support the claim.
Questions for authors

Can restoring sink attention after a spectral ablation rescue the generation? Please also compare the random/non-leading subspace ablation matched for rank and removed activation energy. These are necessary controls that can help distinguish the sink-mediated effects from general representational damage, even though a rigorous causal inference is still needed to claim casualty.

3 comments

Thank you for your thoughtful review and helping us improve the work. We respond to your listed weaknesses and questions below:

  • “manipulating attention sink as proposed in the paper also redistributes attention to other tokens, therefore confounds with the effect of attention sink”: The attention sink is generally thought to function as mechanism for the model to perform a “no-operation” in the case where the input is not relevant to a given attention head. In that case, removing the attention sink should only influence the behaviour of the models because of this redistribution of attention. To test this hypothesis, we performed a modified version of the DAS intervention from Section 4.1,where we set the attention sink value to zero after the softmax is applied. This way, the attention sink is disrupted, but no redistribution takes place. This intervention makes almost no impact on the functioning of both Mistral and Llama with 87% of responses being labelled “Good” (see Section 4.2.1). This is comparable to no perturbation at all. Therefore, this potential confound can be disregarded and does not affect the significance of our results.

  • “The spectral ablation introduces an even stronger confounding, as they can directly remove semantic innformation as well as weakening the sink.”: The interplay between the subspaces that represent semantic information and those that facilitate the attention sink is complex, with one subspace supporting the attention sink at a given layer potentially representing semantic information at future layers (see Figure 5). Disentangling the attention sink and all semantic information on a representational level may be impossible. To help concretely establish the specific role of the attention sink being disabled in both of our spectral perturbation regimes, we more thoroughly experiment with the “Keep 2” variant presented in Figure 5 a) that preserves the top two singular vectors and appears to preserve the attention sink. Results are presented below:

Response TypeBig Upper Keep 2 (n = 3600) FractionSmall Upper Keep 2 (n = 4800) Fraction
Coherent, Irrelevant23.5%2.7%
Relevant, Repetitive41.9%17.3%
Good16.7%78.2%
Bad17.9%1.7%

We can see that the disruption of the attention sink, even temporarily, is essential for both spectral perturbations to work reliably. Small Upper Keep 2 produces nearly 80% Good responses, only 7 points less than the unperturbed models. Meanwhile, Big Upper Keep 2 is very inconsistent in its effect, indicating that the brief disruption of the attention sink demonstrated in Figures 4 and 5 is essential for reliably producing Coherent, Irrelevant responses. These results support the assertion that the disruption of the attention sink in conjunction with specific attention patterns and/or specific underlying representation distributions give rise to the specific output behaviour we see in our spectral perturbations and the investigated jailbreaks.

  • “The claim of "causal intervention" in the paper is too strong”: The interventions we presented in the paper are indeed sufficient causal interventions: the only difference between the usual operation of the model and the results reported are the interventions we conduct. There is not likely to be a necessary condition for producing relevant, repetitive text or coherent, irrelevant text. For instance, an LLM asked to repeat a very repetitive song would produce relevant, repetitive text despite no mechanism failures within the model itself. Finding sufficient conditions to reproduce the phenomena observed in the jailbreaks is an important step in fully characterizing how the transformer architecture can give rise to different types of output.

  • “Can restoring sink attention after a spectral ablation rescue the generation?”: Preventing the attention sink from being disrupted for too many layers in a row can change the impact of an intervention on the model’s generated text. How restoring the attention sink affects generation is impacted by how swiftly the attention sink is restored and the other characteristics of the intervention. Big Upper + 1 slowly restores its attention sink after the spectral ablation, and while the resultant text is generally coherent and irrelevant, it is not completely degenerate or “Bad”. We conducted a similar intervention to DAS from Section 4.1 with only the first layer having its attention sink disrupted. This swift restoration of the attention sink results in 87.5% of generations being “Good”, comparable to no perturbation at all. This underscores the importance of the attention sink in generating uncorrupted text.

  • “Please also compare the random/non-leading subspace ablation matched for rank”: To answer this question, we applied two different perturbations at layers 5,7 and 9 of both models. The first removes either 50% or 70% of singular vectors selected at random, and the second removes the bottom 50% or 70% of singular vectors. These interventions produce “Good” responses 88.5% and 88.6% of the time, respectively. This further speaks to the importance of attention sink specific effects.

We again thank you for your feedback. Please let us know if we can answer any further questions, and if not, please consider increasing your score.

Thanks for the author's effort put into the rebuttal. The additional results are useful, but they still cannot support the causal claim. I understand that disentangling attention sink from semantic representation is challenging, and that is exactly where the gap is. My concern is not whether the interventions causally affect generation, but whether the evidence identifies attention-sink disruption as the mechanism responsible. As the interventions simultaneously alter sink-related structure and potentially semantic information, their effects cannot be attributed specifically to sink disruption without proper controls or mediation evidence. A more rigorous causal inference exercise is necessary to support your "attention sinks cause jailbreak" claim. I therefore decide to keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.