Discourse Boundaries Shape Context Retrieval in Language Models

How do discourse boundaries influence which information language models retrieve from context? We study punctuation and chat-template delimiters using controlled random-token sequences and natural-language prompts. Across tasks, boundary placement strengthens induction and changes the relative influence of competing context associations. We observe these effects across Pythia 2.8B, Qwen 2.5 (1.5B, 3B, 7B base and instruct), Gemma 4 (E2B, E2B-it, E4B, E4B-it), SmolLM2, Falcon 3, and OLMo 2. Mechanistic analysis in GPT-2 small localizes most of the downstream retrieval effect to the contextualized representation of the token immediately following the first occurrence of a repeated cue, carried through a relatively small set of attention-head key/value pathways. Boundary influence decays over a short range of token positions, with distinct decay profiles across attention heads. In natural-language tasks, chat-template boundaries also shift the relative influence of evidence on model predictions. This includes increased effective weighting of post-boundary evidence, even when models natively overweight older information, sometimes by an order of magnitude. Together, these results identify boundary-sensitive attention as a mechanism that shapes access to context and show how discourse formatting can alter the information used by language models.