A Sparse Circuit for Context To Query Information Transfer

A number of head families have been identified for in-context learning (ICL), ranging from heads that promote token and semantic-level copying to heads that perform retrieval and task-level functions. While several of these head families encode dedicated functions that generalize across ICL tasks, less is known about which heads causally mediate information flowbetween the context and the query. Using a causal intervention that patches only the attention connections between query tokens and context tokens, we first identify attention heads that mediate this flow directly. We then identify a shared set of 20 such heads, dubbed ICL20, that supports a diverse range of in-context learning functions. Ablating ICL20 degrades performance across eight diverse evaluation tasks. Moreover, individual ICL20 heads adopt different functional roles by task. Our approach recovers compact, general-purpose circuitry underlying several previously studied ICL functions and reveals two new roles assumed by attention heads during ICL: verbalizing relations between context segments and the query, and using chat-template markers as retrieval anchors.