acceptodds
Under review as a conference paper at ICLR 2027

Textual Salience Tips the Balance: Modulating Long-Context Memory Retrieval in LLMs

Abstract

Reliable retrieval from long contexts is fundamental for LLM agents to reuse interaction histories and prior decisions. Yet even when target information remains available, retrieval can fail when similar memories compete. We ask whether short textual salience cues can shift retrieval priority among competing memories. To isolate this effect from world knowledge and semantic associations, we construct a controlled key–value framework with random bindings and similar-key distractors, evaluating five open-weight LLMs. We test three affectively framed cues—threat, reward, empathy—together with a non-valenced importance cue. Most errors match distractor values already present in context, indicating confusion among competing bindings rather than simple forgetting. Under this interference, well-placed cues improve retrieval accuracy by roughly 30–60 percentage points in most models, although the effect depends on model and placement. Large gains also occur for importance, and no affective family consistently dominates it, indicating a shared textual-salience effect across affectively framed and non-valenced cues. The gains generally do not broadly impair unmarked or updated memories. Internally, target–distractor selection emerges rapidly in mid-to-late layers, where cue-induced differences are strongest; attention shifts toward target values, and MLP interventions at selected layers increase target-value generation probability. Textual salience cues thus act as memory-priority signals that reshape selection among competing memories.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Long-Context Memory Retrieval, Textual Salience

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_por1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether short textual salience cues can influence memory retrieval in long-context LLMs. The authors introduce a controlled key–value retrieval framework with randomly assigned bindings and similar-key distractors to examine how four types of cues—threat, reward, empathy, and importance—affect retrieval accuracy. Experiments across five open-weight LLMs show that retrieval errors primarily arise from selecting competing memories rather than forgetting information. Appropriately placed salience cues improve retrieval accuracy by approximately 30–60 percentage points in most models, with effects varying across models and cue placements. Further mechanistic analyses using Logit Lens, Jacobian Lens, attention analysis, and MLP steering suggest that these cues influence target–distractor selection primarily in mid-to-late layers. These findings highlight the role of textual salience in modulating retrieval priority among competing memories.

Critical strengths
  1. Well-controlled experimental design. The paper introduces a carefully designed experimental framework using random key–value bindings, similar-key distractors, and paired interventions. By keeping memory contents and queries fixed across conditions, the authors effectively isolate the influence of textual salience cues on retrieval performance.

  2. Substantial improvements across multiple models. The experiments demonstrate significant retrieval improvements of approximately 30–60 percentage points in most evaluated models under appropriate marker placements. These results provide convincing evidence that short textual cues can substantially influence memory retrieval under interference.

  3. Balanced experimental analysis. The paper systematically examines different marker families, placements, and model architectures. Importantly, it also reports negative results, particularly the reversed effects observed in Gemma4, providing a more balanced understanding of the conditions under which textual salience cues are effective.

Critical weaknesses

Major Weaknesses

  1. Limited generalization beyond synthetic key–value retrieval. The experiments rely on randomly generated key–value bindings with similar-key distractors. Although this controlled setting effectively isolates retrieval interference, real-world agent memories typically involve richer semantic relationships, contextual dependencies, and varying degrees of relevance. It remains unclear whether the observed improvements generalize to naturalistic long-context retrieval or agent memory scenarios. Additional experiments using more realistic memory retrieval tasks would strengthen the practical significance of the findings.
  2. Insufficient disentanglement of textual salience from lexical and instructional effects. The four marker families are represented by fixed textual expressions that differ in wording, punctuation, and instructional content. Consequently, it is difficult to determine whether the observed improvements arise from textual salience itself or from specific lexical and instructional properties of the markers. Although the paired design controls for memory content and token positions, additional comparisons with neutral markers, paraphrased expressions, or alternative formulations would help clarify the underlying factors responsible for the improvements.
  3. Limited causal evidence for the proposed retrieval mechanism. The mechanistic analyses using Logit Lens, Jacobian Lens, and attention patterns provide useful evidence of target–distractor separation in mid-to-late layers. The MLP steering experiments further provide causal evidence that marker-induced representations can influence retrieval. However, the steering effects are relatively modest, and it remains unclear how much of the overall behavioral improvement these interventions explain. Additional causal analyses, such as activation patching or component-level ablations, would help establish which internal components are necessary for the observed retrieval changes.

Minor Weaknesses

  1. Insufficient explanation of model-dependent effects. While the paper reports that Gemma4 exhibits the opposite trend from the other evaluated models, the underlying reasons for this behavior remain unclear. Further discussion or analysis of potential architectural or training-related factors contributing to these differences would strengthen the interpretation of the results.
  2. Limited statistical uncertainty reporting. The main behavioral experiments use 200 examples per condition for Qwen3-8B and 100 for the other models. Although the observed improvements are substantial in several settings, the behavioral results are primarily reported as point estimates without confidence intervals or statistical significance tests. Reporting uncertainty estimates, particularly for conditions with relatively small performance differences, would help assess the robustness of the findings.
Questions for authors

Have the authors explored what happens when different salience-marker families are assigned to competing memories within the same context? Would one family consistently receive higher retrieval priority, or would the outcome depend primarily on marker placement?

To what extent do the observed improvements depend on the specific wording of the markers? Have the authors considered using neutral markers or paraphrased expressions to distinguish general salience effects from lexical or instructional effects?

The MLP steering experiments demonstrate relatively modest changes in target first-token probability. How much of the behavioral improvement can these interventions explain, and would the effects extend to full-value retrieval accuracy?

11 comments
AuthorsOpenReview ↗

Response to Reviewer por1

W1: Generalization to naturalistic retrieval and agent memory.

We sincerely thank the reviewer for highlighting the importance of evaluating textual salience in realistic memory-retrieval scenarios. We agree that natural-language contexts and agent interaction histories are important directions for establishing the broader applicability of our findings. We apologize that we have not been able to include a sufficiently controlled naturalistic evaluation in the current revision.

Our present framework uses random key–value bindings to isolate competition among memories while minimizing the influence of the model's existing parametric knowledge. Extending this investigation to natural-language settings would require contexts with multiple semantically competing facts, unambiguous target answers, controllable interference, and strictly comparable baseline and marked conditions. We have not identified an existing naturalistic benchmark that simultaneously satisfies these requirements. Constructing and validating such an evaluation would also require carefully separating the effects of textual salience from semantic associations, discourse structure, and recency.

We have explicitly acknowledged this limitation in the revised Section 6 (p. 10, ll. 508–515). We also identify concrete future directions involving realistic agent memory systems with multi-turn interactions, evolving memories, and memory summarization. In these systems, salience cues could be incorporated during memory storage or introduced dynamically at retrieval time, while also evaluating their potential effects on other memories.

We appreciate this suggestion and agree that such evaluations would be a valuable next step toward understanding the practical significance of textual salience in long-context LLMs and agents.

AuthorsOpenReview ↗

Response to Reviewer por1

W2 + Q2: Distinguishing textual salience from lexical and formatting effects.

We sincerely thank the reviewer for suggesting additional controls to distinguish the effects of textual salience from those of specific wording, punctuation, and instructional content. This motivated us to conduct a systematic set of new experiments across all five models.

Specifically, for each of the four marker families, we introduced five token-length-matched control conditions: (1) neutral prose, (2) neutral text retaining the original bracketed format, (3) a salience-preserving paraphrase, (4) punctuation with ordinary spaces, and (5) an ordinary section separator. We fixed the target at the middle position and inserted each variant immediately before the target record (P2). The inserted variants match the original marker's token length and preserve the target-key token position and the substantive downstream token sequence. We also reran the original markers and unmarked baseline on the same examples.

The following table summarizes the mean complete-value retrieval accuracy (%) across the four marker families:

ConditionQwen3Qwen3.5Qwen3.6Gemma4Ministral
No marker63.5057.0067.0063.0028.00
Original marker86.2582.2597.0039.2579.25
Neutral prose69.5043.5072.7549.0071.25
Neutral, same format34.6366.2587.5037.2565.00
Salience paraphrase79.0075.7595.5046.5074.75
Punctuation + spaces77.3865.5083.2539.2535.25
Section separator58.2553.2585.5023.7528.25

These experiments reveal several important findings. First, the original markers outperform all five control types on average in the three Qwen models and Ministral, suggesting that their effects cannot be attributed solely to inserting additional tokens or punctuation. Second, salience-preserving paraphrases retain much of the improvement, indicating that the effect is not restricted to the exact original expressions. Third, neutral and formatting-based insertions can also substantially change retrieval accuracy. For example, neutral prose raises Ministral accuracy from 28.00% to 71.25%, whereas same-format neutral text reduces Qwen3-8B accuracy from 63.50% to 34.63%.

These findings support our interpretation that textual salience influences competitive retrieval, while also revealing contributions from lexical choice, formatting, and context segmentation. We have therefore clarified the scope of this interpretation in the revised manuscript.

The new experiments are summarized in Section 3.2 (p. 5, ll. 242–249) and reported in detail in Appendix D.2.1 (pp. 23–26), including Table 12 (p. 24, ll. 1243–1252), family-specific results (Tables 13–14), and exact control strings (Tables 15–19).

AuthorsOpenReview ↗

Response to Reviewer por1

W3 + Q3: Stronger causal evidence and complete-value retrieval.

We sincerely thank the reviewer for encouraging stronger causal analyses, particularly activation patching and component-level interventions. This suggestion motivated several additional experiments that extend our original MLP steering analysis and provide new insights into the relative contributions of attention and MLP components.

First, we controlled for answer-prefix differences. We repeated the Qwen3-8B attention analysis and performed new component interventions using an identical five-token answer prefix, The corresponding value is , in both conditions. Interventions are applied at the final prefix token, and their outcomes are evaluated using complete-value retrieval accuracy rather than first-token probability alone. This controls generated-prefix content across paired conditions.

Second, we performed cross-sample steering with additional controls. We construct layer-specific directions from paired importance-minus-baseline MLP-output differences using five-fold cross-fitting, so that each test example receives a direction estimated from other examples. The directions are injected into baseline component outputs before residual addition. We compare them against norm-matched random directions and label-flip controls that reverse half of the training paired differences. We subsequently apply the same procedure to attention outputs. We also analyze paired-difference geometry to determine whether averaging cancels useful MLP signals.

Third, we introduced same-example component activation patching. For each paired example, we replace a baseline attention or MLP output with the corresponding output from its own marked condition. Both single-layer and all-layer patching are evaluated, allowing us to compare component effects without relying on cross-example averaged directions.

These experiments lead to three new findings.

  1. Attention interventions produce larger single-layer improvements than MLP interventions. In Qwen3-8B, shared MLP steering improves complete-value accuracy by at most +0.5 pp, while attention steering achieves +2 pp at L23. Same-example attention patching yields +7 pp at L25, correcting 14 baseline errors without introducing new errors, compared with only +1 pp for the best MLP patch. The paired MLP differences remain well aligned after averaging (L18 held-out cosine 0.800; 80.2% norm retention), suggesting that averaging cancellation alone does not explain the weak MLP steering results.

  2. The effective attention layers correspond to the regions identified in our attention maps. In Qwen3-8B, the strongest attention-patching effect occurs at L25, within the mid-to-late-layer region showing increased attention to target values and reduced attention to distractors. In Ministral-3-8B, L24 is both the peak target-value attention-increase layer and the strongest single-layer attention-patching site. This correspondence connects the observational attention patterns with intervention-based retrieval effects.

  3. The benefits of attention interventions are concentrated in selected layers. For Qwen3-8B, all-layer attention patching yields +8 pp, only slightly above the best single-layer gain of +7 pp. Similarly, all-layer attention steering yields +3 pp, compared with +2 pp at L23; both all-layer random and label-flip controls yield −0.5 pp. In Ministral, single-layer attention patching also exceeds MLP patching (+13 vs. +7 pp), although both components reach +15 pp under all-layer patching, highlighting model-dependent differences.

These results are presented in Sections 4.2–4.3 (pp. 8–9, Figures 6–7), with detailed protocols in Appendices E.1 and E.5–E.8 (pp. 42–46) and complete results in Appendices F.2–F.6 (pp. 48–56; Figures 27–34 and Tables 35–43).

Reviewer_zexWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether short textual salience markers change retrieval among competing memories in long-context LLMs. It constructs random key–value retrieval tasks with similar-key distractors and compares paired conditions that preserve memory contents, queries, and downstream token positions. Across five open-weight models, threat, reward, empathy, and non-valenced importance markers produce substantial but placement-dependent retrieval changes. Four models show large improvements in selected conditions, whereas Gemma4 frequently exhibits the opposite effect. Additional experiments examine repeated markers, retrieval of unmarked records, and updated bindings. Mechanistic analyses combine layer-wise vocabulary readouts, source-level attention measurements, and cross-fitted MLP interventions, associating the behavioral differences with mid-to-late-layer target–distractor selection.

Critical strengths
  1. Random bindings reduce reliance on world knowledge, similar-key distractors make error destinations interpretable, and token-position alignment addresses an important confound when inserting text. Independently varying marker and target positions further shows that the effects depend on placement rather than providing a uniform improvement. The large effects in several models make the behavioral finding consequential.
  2. The non-valenced importance cue helps establish that affective framing is not necessary, while Gemma4’s reversed response demonstrates model dependence. The unmarked-target and updating experiments investigate potential costs of prioritization. Cross-fitted interventions also provide evidence beyond observational attention patterns, while avoiding direct transfer of an evaluated example’s own marked activation.
Critical weaknesses
  1. The baseline uses blank placeholders, while all four marker families introduce conspicuous bracketed text and punctuation. Importance is a useful non-affective control, but it remains an explicit salience instruction. Consequently, the shared effect could partly reflect breaking a repetitive sequence, creating a new context segment, or emphasizing subsequent records through formatting. The striking improvement for final targets after middle-marker insertion makes this distinction particularly relevant. Matched neutral text, punctuation-only markers, and ordinary section delimiters are needed to establish what the salience wording contributes.
  2. The principal intervention outcome is target first-token probability, with peak mean gains around 0.02 in several models, rather than complete-value retrieval accuracy. For Qwen3-8B, the median change at the strongest layer is approximately 1.6 × 10⁻⁶, indicating substantial heterogeneity. Moreover, its paired activations are measured after condition-specific generated prefixes, which can introduce differences beyond the input marker. Without norm-matched random directions, shuffled contrasts, common-prefix comparisons, or necessity tests, the steering results show that some transferred activation differences influence output support but do not identify the mechanism responsible for the large behavioral gains.
Questions for authors
  1. Do the main improvements persist relative to token-matched neutral sentences, punctuation-only markers, and ordinary section delimiters? In particular, does salience wording outperform these controls when a middle marker precedes a final target?
  2. Do the MLP interventions improve complete-value exact-match accuracy, particularly on examples where the baseline fails? Please compare against norm-matched random directions and shuffled paired differences, and clarify whether the Qwen3-8B results persist with a common fixed answer prefix.
  3. Can you clarify the large differences between some family-specific no-marker update baselines? For example, Table 17 reports 87% for several families but 0% for empathy in the middle-target, two-update, gap-200 condition. Are these differences caused by independently sampled examples, family-dependent blank placeholders, or another aspect of prompt construction? A shared evaluation set and a common baseline would help assess the update conclusions.
4 comments
AuthorsOpenReview ↗

Response to Reviewer zexW

Thank you for the detailed and constructive review. We have revised the manuscript with additional controls, complete-value interventions, and corrected memory-update results. References below use the printed page and line numbers of the revised PDF.

W1 + Q1: Placement, wording, and formatting.

We sincerely apologize that the original manuscript did not clearly distinguish target positions (Start/Middle/End) from marker insertion sites (P1/P2/P3), causing confusion. P1 precedes the task prompt; P2 immediately precedes the target KV record and moves with that target; P3 follows the memory list before the query. Thus, P2 is target-adjacent even for an End target, not a fixed middle marker. We clarified this in Sec. 2.2 (p. 3, ll. 122–130) and Appendix A (p. 15, ll. 779–798; Table 3).

We added five token-length-matched controls across all four cue families and five models: neutral prose, same-format neutral text, a salience-preserving paraphrase, punctuation with spaces, and a section separator. Inserted variants match the original cue's token length and preserve the target-key token position; the no-marker baseline uses a token-matched blank placeholder. Four-family mean complete-value accuracy (%) for a middle target and P2 is:

ConditionQwen3Qwen3.5Qwen3.6Gemma4Ministral
No marker63.5057.0067.0063.0028.00
Original cue86.2582.2597.0039.2579.25
Neutral prose69.5043.5072.7549.0071.25
Neutral, same format34.6366.2587.5037.2565.00
Salience paraphrase79.0075.7595.5046.5074.75
Punctuation + spaces77.3865.5083.2539.2535.25
Section separator58.2553.2585.5023.7528.25

Original cues outperform all five controls on average across families in the Qwen models and Ministral. Paraphrases retain much of the gain, while neutral and punctuation variants also affect accuracy. We therefore acknowledge contributions from wording and formatting, alongside Gemma4's reversed response. See Sec. 3.2 (p. 5, ll. 243–249) and Appendix D.2.1 (p. 23, ll. 1210–1241; p. 24, Table 12, ll. 1243–1252; exact variants in Tables 15–19, pp. 25–26).

AuthorsOpenReview ↗

Response to Reviewer zexW

W2 + Q2: Controlled component interventions and complete-value retrieval.

We sincerely thank the reviewer for highlighting the limitations of first-token probability and the need for stronger mechanistic controls. These suggestions motivated several additional experiments, revealing further insights into the roles of attention and MLP components in competitive retrieval.

First, we controlled for answer-prefix differences. We repeated the Qwen3-8B attention analysis and conducted component interventions using the same five-token answer prefix, The corresponding value is , in both conditions. Interventions are applied at its final token, and retrieval is evaluated using complete-value accuracy rather than first-token probability alone. This holds answer-prefix content fixed across paired conditions (Sec. 4.2, p. 8, ll. 380–385; Appendix E.1, p. 44, ll. 2343–2352).

Second, we performed controlled cross-sample steering experiments. We extract paired importance-minus-baseline differences from MLP outputs and construct layer-specific shared directions using five-fold cross-fitting, ensuring that each test example receives a direction estimated from other examples. We inject these directions into baseline component outputs before residual addition and compare them with norm-matched random vectors and label-flip controls that reverse half the training paired differences. We apply the same procedure to attention outputs. We also analyze paired-difference geometry to test whether averaging cancels useful MLP signals (Sec. 4.3, p. 8, ll. 414–429; Appendices E.5–E.6, p. 47, ll. 2484–2537; Appendix E.8, p. 48, ll. 2558–2566).

Third, we introduced same-example activation patching to test component effects without averaging. For each paired example, we replace a baseline attention or MLP output with the corresponding output from its own marked condition. We evaluate both single-layer and all-layer replacement to compare the two components directly. These experiments cover Qwen3-8B, Qwen3.5-9B, Ministral-3-8B, and Gemma4-26B-A4B (Sec. 4.3, p. 9, ll. 432–438; Appendix E.7, p. 48, ll. 2538–2556).

These experiments reveal three findings:

  1. Attention interventions produce larger single-layer gains than MLP interventions. In Qwen3-8B, shared MLP steering yields at most +0.5 pp, versus +2 pp for attention steering at L23. Same-example attention patching achieves +7 pp at L25 (14 repaired errors, no new errors), compared with +1 pp for the best MLP patch. MLP differences remain aligned after averaging (L18 held-out cosine 0.800; 80.2% norm retention), indicating that useful shared signals survive averaging.

  2. The effective attention layers align with the attention maps. The strongest Qwen3-8B attention-patching effect occurs at L25, within the mid-to-late region showing increased target-value attention and decreased distractor attention (Fig. 6, p. 8; Fig. 7, p. 9). In Ministral-3-8B, L24 similarly exhibits a strong target-value attention increase and the largest single-layer attention-patching gain of +13 pp. This correspondence connects the observed attention redistribution to intervention-based evidence of its role in retrieval.

  3. Attention contributions exhibit distinct layer-wise patterns across models. In Qwen3-8B, all-layer attention patching yields +8 pp, only slightly above the +7-pp single-layer result, recovering approximately 30% of the full 26.5-pp marker gain. Ministral similarly achieves +13 pp at L24 and +15 pp across all layers, compared with a 16-pp marker gain. In contrast, Qwen3.5-9B and Gemma4-26B-A4B show much larger all-layer effects: attention patching reproduces the full +33-pp improvement in Qwen3.5 and 24 of the 25 pp of accuracy decline in Gemma4. These results reveal both concentrated and distributed attention contributions across models.

The revised findings are presented in Sections 4.2–4.3 (pp. 8–9, Figures 6–7). Experimental protocols appear in Appendices E.1 and E.5–E.8 (pp. 44, 47–48), and complete results are provided in Appendices F.3–F.6 (pp. 53–65), including the cross-model patching comparison in Table 46 (p. 59).

AuthorsOpenReview ↗

Response to Reviewer zexW

Q3: Incorrect memory-update baselines.

We sincerely apologize: the original Table 17 contained evaluation errors. Some failed inference calls returned the literal string connected error, which the scoring script treated as incorrect retrieval, producing the spurious 0% empathy baseline. The examples and no-marker baseline were shared across all four families. A further audit also identified an incorrect 46% entry, which we have corrected.

For Qwen3.5-9B with a middle target, two updates, gap 200, the correct baseline is 87% for every family. Marking the latest binding yields 98% threat, 95% reward, 98% empathy, and 99% importance (correcting the previously reported 46% importance result to 99%). The main text and complete results have been updated: Sec. 3.3 (pp. 6-7, ll. 317-326) and Appendix D.5 (p. 38, ll. 1998-2003; Fig. 19, ll. 2029-2030; Table 28, ll. 2034-2045). We regret these reporting errors and appreciate the reviewer for bringing them to our attention.

AuthorsOpenReview ↗

Follow-up on W2 + Q2: Additional Cross-Model Intervention Results

Dear Reviewer,

Thank you again for your valuable suggestions regarding component interventions and complete-value retrieval. Following our previous response to W2 + Q2, we have further extended the experiments to Qwen3.5-9B and Gemma4-26B, including MLP steering, paired-difference geometry analysis, attention steering, and same-example activation patching.

The additional results reveal an interesting finding: attention-output patching at the answer-prediction position can reproduce nearly the full marker-induced retrieval change in several models. Specifically, applying same-example attention patching across all layers yields:

  • Qwen3.5-9B: +33 percentage points (pp), recovering the full +33-pp marker improvement.
  • Ministral-3-8B: +15 pp, recovering 15 of the 16 pp gained with the marker.
  • Gemma4-26B: −24 pp, reproducing 24 of the 25 pp of accuracy decline induced by the marker.

These results extend our earlier Qwen3-8B findings, where all-layer attention patching recovered approximately 30% of the marker improvement. Interestingly, the strongest single-layer effects already approach the all-layer results in Qwen3-8B and Ministral, whereas Qwen3.5 and Gemma4 exhibit substantially larger changes under all-layer patching.

We also observe clear differences between shared steering and same-example patching. In Qwen3.5 and Gemma4, shared attention steering produces relatively modest changes despite the much larger patching effects. These findings suggest that the influence of attention components on competitive retrieval depends on both the model and the example-specific activation changes.

These additional experiments and findings have now been incorporated into the revised manuscript, with the main results presented in Section 4.3 (pp. 8–9) and detailed comparisons in Appendix F.5 (pp. 59–62, Tables 46–48).

We sincerely appreciate your constructive feedback, which has motivated us to investigate these effects more thoroughly.

Best regards,
The Authors

Reviewer_ZsP2OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses the problem of retrieval from long contexts, for LLM agents to reliably reuse past interactions and decisions in their task execution. To approach studying retrieval issues, the authors propose a measurement framework involving sailence cues, to evaluate if these can shift retrieval priorities among competing memories. The measurement method includes a controlled key-value framework applied to a variety of open-weigth models.

Empirical evidence indicate that salience cues can improve retrieval accuracy in general, with some differences across models, indicating that they act as priority signals that can have a positive influence in selecting appropriate memory fragments, especially in case of "competition" among similar memories.

Critical strengths
  • This work investigates an important problem, that of memory selection during retrieval of in-context information, in the specific case in which interferring memories could harm a proper retrieval. While it is well known that information placed at the incipit of a context, and toward the end of the context are somehow easier to retrieve, information located in between is harder to fetch. Introducing cues seems to alleviate this problem, and improve retrieval performance. This is especially important in an agentic scenario, in which (sub)agents might "receive" a context to execute a task, and information within must be retrieved precisely.
  • In my opinion, the experimental methodology is sound, well executed and well explained. If the authors would have stopped at the end of section 3, I would have been left with many questions, which instead are at least partially answered with the mechanistic analysis in section 4. Thus, although not everything is understood (wh learned about what, where but not exactly why), this paper is well crafted and honest
  • The editorial quality of this paper is excellent: it is easy to follow, it presents ideas clearly, results are discussed and analyzed using appropriate figures, and details are available in the many appendices.
Critical weaknesses
  • My main concern with this work is that we have learned what happens when competing memories are crafted on purpose, we have learned where retrieval problems arise in a transformer architecture, and the role of cues, but we do not know exactly why this happens. Especially in the synthetic scenario built for this study, where there is little ambiguity as to how memory should be indexed (we have unique keys!), why a query fails to properly match a key is not clear to me.
  • As a continuation to the point above, the interference synthetically induced throughout the experiments is not fully explored, to the best of my understanding. One important question that would be "easy" to explore is: can we modulate interference (generate keys that are spanning a range of being very similar to loosely similar to an exsiting memory) and understand if there is a phase transition, a smooth degradation or some other phenomenon affecting retrieval?
  • Finally, one aspect that I find the authors did not elaborate much is: given that cues are helpful (some more than others), what is an operational advice for practitionners to leverage the idea presented in this work to "taint their context" such that it contains such cues? Are the authors alluding at an automatic mechanism that would inject such cues in the context? In other words, I do not understand what can I do now that I know that retrieval interference can be alleviated by cues.
Questions for authors
  1. The proposed experimental framework uses k/v pairs, as this is easier to control and to generate. I wonder how general and robust the conclusion of this work is in more realistic cases in which retrieval is less "deterministic", e.g. considering a different kind of context data, such as text-based narratives.
  2. Related to question 1, in the experimental protocol, I assume that the underlying hypothesis is that the kind of k/v pairs from the context have never been seen before by any of the model under scrutiny: these pairs do not belong to the parametric memory (in other words, the pre-training and SFT data), for otherwise the study of context-based retrieval would be somehow "polluted" by parametric knowledge. So an experiment with more complex data instead of k/v pairs would have to make sure to work on fictional narratives, for example, to make sure they are not present in the model parametric knowledge.

[POST REBUTTAL]

Based on Authors' rebuttal, I have raised my score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ZsP2

We sincerely thank the reviewer for the thoughtful and constructive review, as well as the encouraging assessment of our experimental design, mechanistic analysis, and presentation. We particularly appreciate the insightful suggestions regarding the causes of retrieval failures, the role of key similarity, and the practical applications of our findings in LLM agents. We address each point in detail below.

W1. Why does retrieval fail despite a unique key?

Our experiments show that retrieval errors predominantly select distractor values and that target–distractor selection becomes pronounced in mid-to-late layers. Understanding why such failures occur despite the uniqueness of the queried key has also been a major focus of our recent research. Beyond the analyses presented in this paper, we have actively investigated several possible explanations through complementary experiments:

  1. Numerical precision: On 200 fixed Qwen3-8B examples, retrieval accuracy was 63.0% with BF16 and 64.5% with both FP16 and FP32 computation using the same BF16-origin weights. Increasing arithmetic precision therefore produced only a marginal improvement.
  2. Tokenization and input embeddings: We examined tokenization patterns and embedding similarities between correctly and incorrectly retrieved examples to investigate whether fine-grained key representations contribute to retrieval confusion.
  3. Retrieval-success prediction: We trained a classifier to investigate whether successful and failed retrieval instances exhibit distinguishable features.
  4. Counterfactual interventions: We modified distractor keys and exchanged their associated values to examine how competing bindings influence retrieval decisions.

Despite these extensive investigations, we have not yet established a complete causal explanation for the observed retrieval failures. We believe that understanding why LLMs select incorrect key–value bindings despite the presence of a unique matching key is an important direction for further extending the mechanistic analysis presented in this work.

W2. Effect of key similarity.

Following the reviewer's suggestion, we conducted an additional experiment that fixes the number of distractors while varying the Hamming distance between distractor keys and the target key. A larger Hamming distance indicates lower key similarity. The current Qwen3-8B results are:

Hamming distanceStartMiddleEnd
149.0%59.0%61.0%
272.5%81.5%77.5%
483.5%97.0%95.5%
692.5%98.5%99.5%
898.5%99.0%99.5%
12100.0%99.0%99.0%

The results show a clear relationship between key similarity and retrieval difficulty. At a fixed distractor count, highly similar keys produce substantially lower retrieval accuracy. As Hamming distance increases, accuracy rises rapidly and approaches saturation. This pattern is consistent across all three target positions, with no clear evidence of an abrupt phase transition.

W3. Practical applications in LLM agents.

Our findings suggest two practical directions for incorporating salience cues into agent memory systems.

  1. Salience tagging during memory consolidation. Our experiments show that salience markers placed at the beginning or end of the context can improve retrieval in certain settings and generally do not cause widespread degradation in retrieving other memories (Sections 3.2–3.3). This motivates incorporating lightweight salience tags into memory consolidation and summarization. For example, after completing an interaction, an agent could identify important information based on explicit user feedback or task outcomes and attach priority tags to the corresponding memory records or session summaries. These tags could subsequently accompany the summaries when constructing the context for future interactions, including at the beginning or end.

  2. Dynamic salience tagging at runtime. Our experiments also show that placing a marker immediately before the target memory can substantially improve retrieval accuracy, generally without broad degradation of unmarked memories. In an agent system, a lightweight auxiliary LLM could use the current query, user feedback, and task outcomes to identify relevant candidate memories and dynamically insert salience tags before retrieval. This offers a lightweight way to adjust memory retrieval priority without modifying the underlying memory contents or model parameters.

AuthorsOpenReview ↗

Response to Reviewer ZsP2: Questions

Q1–Q2. Generalization to naturalistic retrieval and parametric knowledge.

We sincerely thank the reviewer for raising these important questions and for the valuable suggestion of using fictional narratives to reduce the influence of parametric knowledge. We fully agree that extending our investigation to natural-language contexts and realistic memory scenarios would be valuable for understanding the broader applicability of textual salience cues.

Previous studies have demonstrated that long-context retrieval difficulties also arise in natural-language settings. For example, Lost in the Middle (Liu et al., 2024) reveals substantial positional sensitivity in both multi-document question answering and key–value retrieval. Similarly, NoLiMa (Modarressi et al., 2025) investigates long-context retrieval when queries and relevant information lack direct lexical overlap, highlighting additional challenges introduced by semantic associations and indirect matching. These findings establish the broader relevance of long-context retrieval difficulties and motivate further investigation of how textual salience might influence retrieval in naturalistic settings.

However, we have not yet identified a natural-language dataset that satisfies the experimental requirements of our study. Our investigation focuses specifically on how textual salience cues influence retrieval when multiple similar memories compete. To examine this question rigorously, a suitable naturalistic benchmark would need to satisfy several requirements: (1) sufficiently long and coherent natural-language contexts; (2) clearly identifiable target facts and unambiguous ground-truth answers; (3) competing memories whose similarity and interference strength can be systematically controlled; and (4) minimal contamination from the model's existing parametric knowledge. Furthermore, the baseline and marked conditions must remain strictly comparable, including memory content, queries, and token-position alignment.

We particularly appreciate the reviewer's suggestion of constructing fictional narratives with previously unseen facts. Such an approach could effectively reduce interference from memorized factual knowledge and would be a promising direction for extending our experiments. Nevertheless, constructing sufficiently long and coherent fictional narratives while introducing numerous competing memories with controlled similarity presents additional challenges. Natural-language narratives also contain semantic relationships, discourse structures, and contextual associations that may independently influence retrieval. Carefully controlling these factors, while ensuring that the target answers remain unambiguous, would require substantial dataset construction and validation.

Given these requirements, we have not been able to complete a sufficiently controlled naturalistic evaluation within the current rebuttal period. We believe that a carefully constructed and validated benchmark would provide more informative evidence than a small-scale experiment without adequate control of these factors.

These considerations also motivate our use of randomly generated key–value bindings in the present study. Random bindings establish previously unseen associations between keys and values, substantially reducing the influence of parametric knowledge. They also allow us to precisely control the number and similarity of competing memories and determine whether an incorrect response corresponds to a specific distractor value. Together with our strictly paired intervention design, this enables a focused investigation of how textual salience cues affect retrieval priority under controlled memory competition.

We greatly appreciate the reviewer's suggestions and consider naturalistic retrieval an important next step. In particular, constructing long-form fictional narratives with controlled competing facts and evaluating salience cues in realistic agent interaction histories would provide a valuable extension of the present work.

AuthorsOpenReview ↗

Response to Reviewer ZsP2

Update: Additional experiments incorporated into the revised manuscript

We sincerely thank the reviewer again for the constructive suggestions. We would like to highlight several updates now incorporated into our revised manuscript.

Following the reviewer's suggestion, we have added the key-similarity experiment to Section 3.1 (p. 4, ll. 169–172) and Appendix D.1.1 (p. 21, Table 8). By varying the Hamming distance while keeping the distractor configuration fixed, we find that retrieval accuracy increases sharply as key similarity decreases, approaching saturation across all three target positions.

We have also strengthened the mechanistic analysis with additional attention and MLP activation-patching and steering experiments, reported in Sections 4.2–4.3 (pp. 8–9, Figures 6–7). These experiments provide further evidence of the role of attention components in competitive retrieval.

Finally, we have expanded the discussion of practical applications in Section 6 (p. 10, ll. 508–515), including salience tagging during memory storage and dynamic cue insertion at retrieval time in realistic agent memory systems.

We greatly appreciate the reviewer's valuable suggestions, which have helped us strengthen the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.