acceptodds
Under review as a conference paper at ICLR 2027

Discourse Boundaries Shape Context Retrieval in Language Models

Abstract

How do discourse boundaries influence which information language models retrieve from context? We study punctuation and chat-template delimiters using controlled random-token sequences and natural-language prompts. Across tasks, boundary placement strengthens induction and changes the relative influence of competing context associations. We observe these effects across Pythia 2.8B, Qwen 2.5 (1.5B, 3B, 7B base and instruct), Gemma 4 (E2B, E2B-it, E4B, E4B-it), SmolLM2, Falcon 3, and OLMo 2. Mechanistic analysis in GPT-2 small localizes most of the downstream retrieval effect to the contextualized representation of the token immediately following the first occurrence of a repeated cue, carried through a relatively small set of attention-head key/value pathways. Boundary influence decays over a short range of token positions, with distinct decay profiles across attention heads. In natural-language tasks, chat-template boundaries also shift the relative influence of evidence on model predictions. This includes increased effective weighting of post-boundary evidence, even when models natively overweight older information, sometimes by an order of magnitude. Together, these results identify boundary-sensitive attention as a mechanism that shapes access to context and show how discourse formatting can alter the information used by language models.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, in-context retrieval, attention mechanism, recency bias, induction heads

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wZjjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how punctuation and chat-turn boundaries affect retrieval from context. In random-token sequences, boundaries change the probability of retrieving a previously observed continuation, with effects depending on their placement relative to the evidence and query. Path patching in GPT-2 small attributes much of the effect to key/value pathways at the continuation token. The paper also tests 14 natural-language domains and finds that separating two facts with an empty assistant turn generally shifts predictions toward the second fact.

Critical strengths

The synthetic experiments are well controlled. Replacing filler tokens preserves sequence length and the positions of the cue and target. Sweeping the boundary across the sequence is particularly useful. It separates an effect near the stored association from an effect near the retrieval query. This provides more information than a comparison between two prompt templates.

The GPT-2 patching results go beyond an attention visualization. Jointly patching keys and values at the cue and continuation positions restores about 85% of the effect, while the reverse intervention removes about 84%. The strong interaction between key and value channels is also interesting. These results give a reasonably concrete account of where the effect is carried in this setting.

Critical weaknesses

I am not sure the GPT-2 experiments explain what happens in the natural-language tasks. Putting the second fact in a new user turn adds tokens and changes how the conversation reads. The model might interpret it as a correction or update to the first fact. The random-token experiments do not rule out this explanation. Although the authors acknowledge that the circuit analysis is limited to GPT-2, the discussion sometimes treats it as an explanation for both sets of results. A control with the same token length would help, as would testing whether intervening on the proposed mechanism changes the natural-language effect in an instruction-tuned model.

The natural-language results suggest a real shift toward the second fact, but some claims about its size seem too strong. In Figure 6, dividing by a very small positive single-fact gap can make the normalized effect look large. The ratio in Appendix A.7 has a similar problem when the single-turn order effect is near zero. It also uses absolute values, so it does not distinguish a stronger preference for the first fact from a stronger preference for the second. Finally, a normalized swing of one does not necessarily mean the model answers as though it saw only the second fact. To support that claim, the authors should compare the actual predictions with the second-fact-only predictions. Showing the raw changes alongside the normalized results would make the “tenfold” claim easier to assess.

Questions for authors

Does the natural-language effect remain when the single-turn control contains an equal-length separator, or when the intervening assistant turn contains a neutral acknowledgment? These comparisons would help distinguish a local boundary effect from conversational update behavior.

Can you report the raw single- and split-turn swings, the single-fact gaps, and uncertainty across fact-pairs? In particular, does Figure 6 retain its pattern after excluding small positive denominators, and do the individual split-turn predictions actually approach their corresponding single-fact predictions?

Which experimental setup produced Figure 2: the one described in Section 4.1 or Appendix A.2?

2 comments
AuthorsOpenReview ↗

Response to Reviewer wZjj

Thank you for taking the time to craft such a thoughtful review. We enjoyed reading your suggestions and worked hard to incorporate them into our paper.

New Experiments: We conducted a series of new experiments that more granularly defines and tests boundaries, including your suggestions of an equal-length separator and a neutral acknowledgment from the assistant turn. There are four single-turn variants: the two facts are separated by (1) a single space, (2) the word 'and', (3) a new sentence and (4) a new line. There are three split-turn variants: the assistant turn is (1) blank, (2) replies 'Okay.' and (3) acknowledges the first fact. Across all four single-turn variants, the amplification of the more recent fact was significantly smaller than across the three split-turn variants. These results help to confirm that the natural language results hold even to perturbations of boundary types within a single turn, when compared to different assistant turns across two different turns. We ran this on three representative configurations spanning three families and both training types (Qwen 2.5 1.5B-Instruct, Gemma 4 E2B-it, and OLMo 2 1B), across all 14 domains, and included it in our main manuscript with figures that portray the conditions and results.

New Dependent Variable: We have changed the way we measure amplification to address your concerns around dividing by small, positive gaps. Instead, we now normalize E(LH) - E(HL) by the 'bucket range', or the difference in the maximum and minimum answers for a specific domain. This allows for a dependent variable that is both interpretable (what percent of the feasible range is shifted by switching fact order?) and robust to skewness (it is a constant fixed for each domain and doesn't vary by model). The core of our results (split-turns increase swing from switching fact order) holds under this new framing, as is best depicted in Figure 6. However, we also included the single- and split-turn swings and single-fact gaps in the Appendix per your request (even though the main manuscript now uses the 'bucket range'). The vast majority of rows see a larger 'Split swing' than a 'Single swing'.

Experimental Setup for original Figure 2: We thank the reviewer for this catch; the settings used for Fig. 2 were what was described in Appx. A.2 (|F1|=32, |F2|=128, |K|=1), not what we had written in Sec. 4.1 (|F1|=|F2|=32, |K|=3). We have corrected the text.

Thanks the authors for taking efforts to address most of my concerns. I am willing to raise my score to 3.

Reviewer_vvkwOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies (and mechanistically interprets) how discourse boundary shape context retrieval.

Critical strengths
  • This paper has done a good job in finding the intersection of discourse boundaries and how they shape the LLM generation.
  • The paper has evaluated a good number of models. The results support the paper's claim.
  • The paper also goes forward to mechanistic interpretability to check the models' internal computation and explain the models' behaviors.
Critical weaknesses

Major: The primary weakness of this manuscript should be a lack of baselines. The proposal is how discourse helps shape the structure for context representation, and we do have quite a few discourse-aware works for RAG:

I am not suggesting the authors should replicate them, since the purpose is different. However, I am wondering if they can borrow their structures and incorporate them into the author's system and compare? It is fine if it is out of scope of the paper -- so please help distinguish this manuscript from the prior papers in ACL and LREC.

Minor: For mechanistic interpretability, the authors might already know a few related works. Consider citing them and having an in-depth discussion about the differences with them:

Questions for authors
  • Are there any actionable takeaways that LLMs practitioners can use from your identified mechanistic components?
  • AI declaration section: I am wondering if you used any specific AI tools to create the figures? They are very nice, but I am not sure if they are disclosed in the section.
1 comment
AuthorsOpenReview ↗

Response to Reviewer vvkw

Thank you for your thoughtful review of our paper. We enjoyed reading your comments and insights, and have worked hard to address them in our paper. More detailed comments are below.

Disco-RAG+DiscoRAG: We agree that our findings might connect with discourse-structure-aware RAG by informing which structures are most salient for LM processing. We have added some thoughts to this end in the discussion, and appreciate the suggested connection.

Related Mechanistic Interpretability Works: We thank the reviewer for pointing us toward this work that takes a mech interp lens to the processing of discourse structure more broadly than our current boundary-focused submission. We think there are exciting directions to pursue at the intersection! We have added these to our related work.

Actionable takeaways: One of the takeaways relates to the growing understanding that LMs process the same semantic content differently based on these structural cues. Our work can be seen as a contribution to a specific way to influence processing: add louder discourse boundaries to improve salience of what comes after. Another is related to your RAG point above: with more insight about how different discourse boundaries signal salience, we might parse discourse structure differently to best respect the LM processing. In addition to the discussion around the RAG point, we've also added references around the point of structure being important even with semantic content fixed, including Sclar et al. 2024 (https://openreview.net/forum?id=RIu5lyNXjT) and Seleznyov et al. 2025 (https://aclanthology.org/2025.findings-emnlp.1109/).

AI figures: Thank you for the comment. For several of the figures, we adopted a hybrid approach where a coding agent provided a first pass and then we modified ourselves. Our original AI Use statement includes "Additionally, we used generative AI tools to draft and edit prose, create figures and captions, and write and organize code." Please let us know if you have a specific edit in mind.

Reviewer_fRvTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores how discourse boundaries (such as punctuation and chat template turn markers) influence the way language models extract information from context. Through controlled randomized token experiments, natural language reasoning tasks based on multiple open-source models, and a mechanistic analysis of GPT-2 small, the authors demonstrate that these boundaries enhance inductive capabilities and rebalance competing information, with dialogue turn boundaries significantly amplifying the impact of facts introduced in the latter half of a conversation.

Critical strengths

S1. The experimental design is rigorously controlled. In Section 4.1, the authors designed a randomized token experiment that kept the content, position, and sequence length constant while varying only the boundary conditions, thereby clearly isolating the effect of boundaries.

S2. The study demonstrates a certain breadth and practical significance. The authors validated the effect of discourse boundaries across multiple model families, including baseline models and fine-tuned models. The findings of this study also have practical implications: in chat scenarios, presenting the same facts all at once in a single dialogue turn, rather than spreading them across multiple turns, significantly alters the model’s responses, which is of great significance for the design and evaluation of prompts.

Critical weaknesses

W1. The motivation should better highlight what is non-obvious. The research question the authors target, namely that punctuation and discourse structure affect information processing, is largely expected: segmenting content is a core function of punctuation, and LMs are trained on text that follows these conventions. However, the authors’ experiments also reveal the direction in which boundary placement affects retrieval (e.g., a boundary makes the following content more retrievable later), which I find the more interesting finding. I suggest the authors discuss this further, for instance why this direction arises and how it compares to intuitive expectations, and reframe the introduction around it.

W2. The natural-language experiment does not isolate the discourse boundary. In Fig. 1, both conditions already separate the two facts with a sentence boundary (a period and a newline). The comparison is therefore between a sentence boundary and a turn boundary, rather than between boundary and no boundary. The split-turn condition also changes conversational pragmatics (the latest user turn may be prioritized), length, and recency at the same time. Applying the controlled design of Sec. 4.1 here might help clarify this, for example moving the boundary to before the first fact.

W3. Minor (presentation). The font sizes in all figures are very small, which makes them hard to read. Fig. 6 is particularly difficult to interpret: many model labels overlap in the dense cluster around the diagonal and are essentially illegible. I suggest enlarging the fonts throughout, and for Fig. 6, using markers with a legend (or a table in the appendix) instead of overlapping text labels.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer fRvT

Thank you for taking the time to review our paper. We worked hard to address your concerns, and the changes are detailed below.

W1. We agree that the concept that punctuation/boundaries affect processing is well-established, and have revised the text in several places to more clearly emphasize the two non-obvious highlights. First, boundaries cause both a boost in forward salience and backward retrieval, as shown by our synthetic sequences across a range of models and boundaries. We find it surprising that both effects emerge from the boundaries, and have attempted to shed light on the specific mechanisms at play (eg the keys and values both carry effects in the tokens immediately following a boundary, Fig. 4). Second, it's not obvious how similar a chat template is to other discourse boundaries; we quantify the effect across models and find that the difference is predominantly categorical. To better support this second finding, we have added new natural language results that vary the boundary systematically for three models; please see the next item for more details.

W2. We conducted a series of new experiments that more granularly defines and tests boundaries. There are four single-turn variants: the two facts are separated by (1) a single space, (2) the word 'and', (3) a new sentence and (4) a new line. There are three split-turn variants: the assistant turn is (1) blank, (2) replies 'Okay.' and (3) acknowledges the first fact. Across all four single-turn variants, the amplification of the more recent fact was significantly smaller than across the three split-turn variants. These results help to confirm that the natural language results hold even to perturbations of boundary types within a single turn, when compared to different assistant turns across two different turns. We ran this on three representative configurations spanning three families and both training types (Qwen 2.5 1.5B-Instruct, Gemma 4 E2B-it, and OLMo 2 1B), across all 14 domains, and included it in our main manuscript with figures that portray the conditions and results.

W3. We have redesigned the original Fig. 6 around readability: instead of each point being a model name, each point is a shape that connects to a model family. In addition to zooming in, we believe this has made the figure easier to understand. For Figs. 2 and 3, we increased the size of axes, titles and labels.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Changes

We are thankful for the reviewers' thoughtful comments, and are glad they recognized both the potential impact of our work and appreciated the mix of synthetic and natural language experiments. We have worked hard to incorporate their suggestions into our manuscript, and believe the work has been greatly improved due to their input. The high-level changes are as follows:

  • We conducted a series of new natural language experiments that more granularly define and test boundaries. The results, across four single-turn variants (e.g., facts are separated by a space, a word, a new line or a new sentence) and three split-turn variants (e.g., the assistant turn is empty, contains a neutral response, or a specific acknowledgement), confirm that the amplification of the more recent fact is significantly smaller when the facts are presented in a single turn.

  • We have revised our framing and discussion to emphasize the most surprising, novel findings. Specifically, boundaries have a dual role of boosting forward salience and backwards retrieval, and chat templates tend to have a categorically different effect on retrieval of post-template information.

  • We have edited existing figures (and new figures) in line with the review team's request for readability.

We also respond in detail to each reviewer in the rest of the form, and uploaded updated supplementary material to reflect our new experiments. Please do not hesitate to reach out with any other questions or suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.