Sink Before You Speak: Connecting Language Generation, Attention Patterns, and Representational Geometry

Large Language Models (LLMs) have demonstrated an impressive capability to generate coherent text that is relevant to the subject matter provided in a user's prompt. Despite their sophistication, LLMs have also been shown to generate completely irrelevant text or strings of repeating, relevant tokens both in naturalistic settings and through adversarially constructed prompts in natural language. While some research has been done into the mechanistic cause for these sorts of generations, these investigations provide incomplete explanations on account of considering these phenomena independently. In this work, we provide a unified account for how both relevant, repetitive and coherent, irrelevant generations arise from specific activation patterns in the attention mechanisms of these models. This explanation centers on the attention sink, the emergent behaviour where high softmax attention values are assigned to seemingly mundane tokens such as the start-of-sentence delimiter. We evidence this explanation by directly modifying the attention maps and performing a novel perturbation to the latent representations. These causal interventions not only reliably induce the types of language generation we are interested in, but also the attention patterns we hypothesize to be at fault for them. We finally use this per-prompt spectral approach to characterize the operation of the attention sink. The findings of this work give a more complete understanding of the operation of LLMs, as well as how models that are seemingly steeled against existing jailbreaks remain susceptible to the shortcomings of the transformer architecture.