- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether short textual salience cues can influence memory retrieval in long-context LLMs. The authors introduce a controlled key–value retrieval framework with randomly assigned bindings and similar-key distractors to examine how four types of cues—threat, reward, empathy, and importance—affect retrieval accuracy. Experiments across five open-weight LLMs show that retrieval errors primarily arise from selecting competing memories rather than forgetting information. Appropriately placed salience cues improve retrieval accuracy by approximately 30–60 percentage points in most models, with effects varying across models and cue placements. Further mechanistic analyses using Logit Lens, Jacobian Lens, attention analysis, and MLP steering suggest that these cues influence target–distractor selection primarily in mid-to-late layers. These findings highlight the role of textual salience in modulating retrieval priority among competing memories.
Critical strengths
-
Well-controlled experimental design. The paper introduces a carefully designed experimental framework using random key–value bindings, similar-key distractors, and paired interventions. By keeping memory contents and queries fixed across conditions, the authors effectively isolate the influence of textual salience cues on retrieval performance.
-
Substantial improvements across multiple models. The experiments demonstrate significant retrieval improvements of approximately 30–60 percentage points in most evaluated models under appropriate marker placements. These results provide convincing evidence that short textual cues can substantially influence memory retrieval under interference.
-
Balanced experimental analysis. The paper systematically examines different marker families, placements, and model architectures. Importantly, it also reports negative results, particularly the reversed effects observed in Gemma4, providing a more balanced understanding of the conditions under which textual salience cues are effective.
Critical weaknesses
Major Weaknesses
- Limited generalization beyond synthetic key–value retrieval. The experiments rely on randomly generated key–value bindings with similar-key distractors. Although this controlled setting effectively isolates retrieval interference, real-world agent memories typically involve richer semantic relationships, contextual dependencies, and varying degrees of relevance. It remains unclear whether the observed improvements generalize to naturalistic long-context retrieval or agent memory scenarios. Additional experiments using more realistic memory retrieval tasks would strengthen the practical significance of the findings.
- Insufficient disentanglement of textual salience from lexical and instructional effects. The four marker families are represented by fixed textual expressions that differ in wording, punctuation, and instructional content. Consequently, it is difficult to determine whether the observed improvements arise from textual salience itself or from specific lexical and instructional properties of the markers. Although the paired design controls for memory content and token positions, additional comparisons with neutral markers, paraphrased expressions, or alternative formulations would help clarify the underlying factors responsible for the improvements.
- Limited causal evidence for the proposed retrieval mechanism. The mechanistic analyses using Logit Lens, Jacobian Lens, and attention patterns provide useful evidence of target–distractor separation in mid-to-late layers. The MLP steering experiments further provide causal evidence that marker-induced representations can influence retrieval. However, the steering effects are relatively modest, and it remains unclear how much of the overall behavioral improvement these interventions explain. Additional causal analyses, such as activation patching or component-level ablations, would help establish which internal components are necessary for the observed retrieval changes.
Minor Weaknesses
- Insufficient explanation of model-dependent effects. While the paper reports that Gemma4 exhibits the opposite trend from the other evaluated models, the underlying reasons for this behavior remain unclear. Further discussion or analysis of potential architectural or training-related factors contributing to these differences would strengthen the interpretation of the results.
- Limited statistical uncertainty reporting. The main behavioral experiments use 200 examples per condition for Qwen3-8B and 100 for the other models. Although the observed improvements are substantial in several settings, the behavioral results are primarily reported as point estimates without confidence intervals or statistical significance tests. Reporting uncertainty estimates, particularly for conditions with relatively small performance differences, would help assess the robustness of the findings.
Questions for authors
Have the authors explored what happens when different salience-marker families are assigned to competing memories within the same context? Would one family consistently receive higher retrieval priority, or would the outcome depend primarily on marker placement?
To what extent do the observed improvements depend on the specific wording of the markers? Have the authors considered using neutral markers or paraphrased expressions to distinguish general salience effects from lexical or instructional effects?
The MLP steering experiments demonstrate relatively modest changes in target first-token probability. How much of the behavioral improvement can these interventions explain, and would the effects extend to full-value retrieval accuracy?
11 comments
Response to Reviewer por1
W2 + Q2: Distinguishing textual salience from lexical and formatting effects.
We sincerely thank the reviewer for suggesting additional controls to distinguish the effects of textual salience from those of specific wording, punctuation, and instructional content. This motivated us to conduct a systematic set of new experiments across all five models.
Specifically, for each of the four marker families, we introduced five token-length-matched control conditions: (1) neutral prose, (2) neutral text retaining the original bracketed format, (3) a salience-preserving paraphrase, (4) punctuation with ordinary spaces, and (5) an ordinary section separator. We fixed the target at the middle position and inserted each variant immediately before the target record (P2). The inserted variants match the original marker's token length and preserve the target-key token position and the substantive downstream token sequence. We also reran the original markers and unmarked baseline on the same examples.
The following table summarizes the mean complete-value retrieval accuracy (%) across the four marker families:
| Condition | Qwen3 | Qwen3.5 | Qwen3.6 | Gemma4 | Ministral |
|---|---|---|---|---|---|
| No marker | 63.50 | 57.00 | 67.00 | 63.00 | 28.00 |
| Original marker | 86.25 | 82.25 | 97.00 | 39.25 | 79.25 |
| Neutral prose | 69.50 | 43.50 | 72.75 | 49.00 | 71.25 |
| Neutral, same format | 34.63 | 66.25 | 87.50 | 37.25 | 65.00 |
| Salience paraphrase | 79.00 | 75.75 | 95.50 | 46.50 | 74.75 |
| Punctuation + spaces | 77.38 | 65.50 | 83.25 | 39.25 | 35.25 |
| Section separator | 58.25 | 53.25 | 85.50 | 23.75 | 28.25 |
These experiments reveal several important findings. First, the original markers outperform all five control types on average in the three Qwen models and Ministral, suggesting that their effects cannot be attributed solely to inserting additional tokens or punctuation. Second, salience-preserving paraphrases retain much of the improvement, indicating that the effect is not restricted to the exact original expressions. Third, neutral and formatting-based insertions can also substantially change retrieval accuracy. For example, neutral prose raises Ministral accuracy from 28.00% to 71.25%, whereas same-format neutral text reduces Qwen3-8B accuracy from 63.50% to 34.63%.
These findings support our interpretation that textual salience influences competitive retrieval, while also revealing contributions from lexical choice, formatting, and context segmentation. We have therefore clarified the scope of this interpretation in the revised manuscript.
The new experiments are summarized in Section 3.2 (p. 5, ll. 242–249) and reported in detail in Appendix D.2.1 (pp. 23–26), including Table 12 (p. 24, ll. 1243–1252), family-specific results (Tables 13–14), and exact control strings (Tables 15–19).
Response to Reviewer por1
W3 + Q3: Stronger causal evidence and complete-value retrieval.
We sincerely thank the reviewer for encouraging stronger causal analyses, particularly activation patching and component-level interventions. This suggestion motivated several additional experiments that extend our original MLP steering analysis and provide new insights into the relative contributions of attention and MLP components.
First, we controlled for answer-prefix differences. We repeated the Qwen3-8B attention analysis and performed new component interventions using an identical five-token answer prefix, The corresponding value is , in both conditions. Interventions are applied at the final prefix token, and their outcomes are evaluated using complete-value retrieval accuracy rather than first-token probability alone. This controls generated-prefix content across paired conditions.
Second, we performed cross-sample steering with additional controls. We construct layer-specific directions from paired importance-minus-baseline MLP-output differences using five-fold cross-fitting, so that each test example receives a direction estimated from other examples. The directions are injected into baseline component outputs before residual addition. We compare them against norm-matched random directions and label-flip controls that reverse half of the training paired differences. We subsequently apply the same procedure to attention outputs. We also analyze paired-difference geometry to determine whether averaging cancels useful MLP signals.
Third, we introduced same-example component activation patching. For each paired example, we replace a baseline attention or MLP output with the corresponding output from its own marked condition. Both single-layer and all-layer patching are evaluated, allowing us to compare component effects without relying on cross-example averaged directions.
These experiments lead to three new findings.
-
Attention interventions produce larger single-layer improvements than MLP interventions. In Qwen3-8B, shared MLP steering improves complete-value accuracy by at most +0.5 pp, while attention steering achieves +2 pp at L23. Same-example attention patching yields +7 pp at L25, correcting 14 baseline errors without introducing new errors, compared with only +1 pp for the best MLP patch. The paired MLP differences remain well aligned after averaging (L18 held-out cosine 0.800; 80.2% norm retention), suggesting that averaging cancellation alone does not explain the weak MLP steering results.
-
The effective attention layers correspond to the regions identified in our attention maps. In Qwen3-8B, the strongest attention-patching effect occurs at L25, within the mid-to-late-layer region showing increased attention to target values and reduced attention to distractors. In Ministral-3-8B, L24 is both the peak target-value attention-increase layer and the strongest single-layer attention-patching site. This correspondence connects the observational attention patterns with intervention-based retrieval effects.
-
The benefits of attention interventions are concentrated in selected layers. For Qwen3-8B, all-layer attention patching yields +8 pp, only slightly above the best single-layer gain of +7 pp. Similarly, all-layer attention steering yields +3 pp, compared with +2 pp at L23; both all-layer random and label-flip controls yield −0.5 pp. In Ministral, single-layer attention patching also exceeds MLP patching (+13 vs. +7 pp), although both components reach +15 pp under all-layer patching, highlighting model-dependent differences.
These results are presented in Sections 4.2–4.3 (pp. 8–9, Figures 6–7), with detailed protocols in Appendices E.1 and E.5–E.8 (pp. 42–46) and complete results in Appendices F.2–F.6 (pp. 48–56; Figures 27–34 and Tables 35–43).
Response to Reviewer por1
W1: Generalization to naturalistic retrieval and agent memory.
We sincerely thank the reviewer for highlighting the importance of evaluating textual salience in realistic memory-retrieval scenarios. We agree that natural-language contexts and agent interaction histories are important directions for establishing the broader applicability of our findings. We apologize that we have not been able to include a sufficiently controlled naturalistic evaluation in the current revision.
Our present framework uses random key–value bindings to isolate competition among memories while minimizing the influence of the model's existing parametric knowledge. Extending this investigation to natural-language settings would require contexts with multiple semantically competing facts, unambiguous target answers, controllable interference, and strictly comparable baseline and marked conditions. We have not identified an existing naturalistic benchmark that simultaneously satisfies these requirements. Constructing and validating such an evaluation would also require carefully separating the effects of textual salience from semantic associations, discourse structure, and recency.
We have explicitly acknowledged this limitation in the revised Section 6 (p. 10, ll. 508–515). We also identify concrete future directions involving realistic agent memory systems with multi-turn interactions, evolving memories, and memory summarization. In these systems, salience cues could be incorporated during memory storage or introduced dynamically at retrieval time, while also evaluating their potential effects on other memories.
We appreciate this suggestion and agree that such evaluations would be a valuable next step toward understanding the practical significance of textual salience in long-context LLMs and agents.