acceptodds
Under review as a conference paper at ICLR 2027

Attention Guided Policy Learning for Context Selection in Long Horizon Agents

Abstract

Large language models increasingly serve as the reasoning core of agents that operate over long, multi-turn interactions. Because every new observation is ap- pended to a running context, the context such systems present to the model grows far faster than it is used, and an increasing share of it is irrelevant to the query at hand. Recent works show that this bloat is not a purely computational nuisance: accuracy degrades sharply and non-linearly as soon as distractors enters the con- text, with most of the damage done long before the window is full. Compressing or filtering context with signals external to the target model, such as an auxiliary scorer’s perplexity or embedding similarity, discards a resource the model already computes internally: a small set of attention heads and activation directions has been shown to track, with high fidelity, which tokens are being retrieved, copied, or bound into an answer. We formulate the problem of learning a context-curation policy that reads this internal, attention-level relevance signal directly from the ex- ecuting model and combines it with an outcome-grounded reinforcement-learning objective to select the subset of an agent’s accumulated context that should be forwarded to the next inference call. We propose CURATE, which learns this pol- icy with reinforcement learning, assigning credit for selection decisions over long horizons from sparse task-level reward. CURATE learns to identify and discard distractors without any relevance labels, retaining less than 10% of the context while preserving downstream accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
context-selection, LLM representation, long-horizon agents

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_9EPAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a context selection framework CURATE based on reinforcement learning for long-term sequence LLM agents. This method first identifies a set of fixed retrieval heads from the frozen LLM and uses their attention patterns to represent the relevance of the input context segments. Subsequently, a lightweight MLP or CNN strategy makes decisions on whether to retain or discard each segment. This strategy is trained using relative policy gradients within the group and uses answer correctness and memory cost as evaluation metrics, without the need for explicit relevance labels.

Critical strengths
  1. An interesting method that utilizes internal model signals for context selection.
  2. It has strong empirical results and practical analysis.
Critical weaknesses
  1. It is not clear whether there are probe questions during the online scenario selection process.
  2. The contribution of the optimization method based on reinforcement learning compared to the heuristic method based on attention has not been fully separated.
Questions for authors

Q1. Probe Question Availability.

During the training and evaluation processes, it is important to understand how the probing questions are sampled, especially for LoCoMo. Does the selection strategy assess the standard benchmark questions that are only raised in the later stages of the conversation? Additionally, how does this method identify which questions in the current memory have sufficient evidence without relying on gold evidence annotations?

Q2. Contribution of RL-based Context Selection.

Could the author use the same retrieval head, question formulation, and memory budget to compare CURATE with a simple attention-based ranking baseline? Since Table 1 already shows a significant separation between the gold segments and the interfering segments, such a comparison would help demonstrate the additional advantages brought by learning the selection strategy through reinforcement learning.

Q3. End-to-end Computational Efficiency.

The reported delay of 1.6 seconds per query in Table 5 includes whether it encompasses the repeated detection generation and attention extraction forward propagation required during the memory construction process? Can the authors report the complete scene-level or end-to-end costs, separately explaining the expenses for feature extraction, policy inference, and final answer generation?

Q4. Baseline Comparison and Generalization.

Memory-R1 [1] is a memory management method based on reinforcement learning, which has been evaluated on LoCoMo and LongMemEval. Can the authors discuss its relationship with CURATE and provide a comparative analysis under the condition-matching setting (if feasible)?

In addition, CURATE retains 50.1% of the context on LongMemEval, while LoCoMo only retains 4.3%. Can the authors further analyze this difference and evaluate this method in the context of natural occurrence interference items beyond the synthetic generation process?

Reference

[1] Yan et al., "Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning." ACL 2026. https://aclanthology.org/2026.acl-long.583/

9 comments
AuthorsOpenReview ↗

Part 1: Response (1/3)

We sincerely thank the reviewer for their thoughtful feedback and constructive evaluation of our work.

W1, Q1 (Probe Question Availability) :- Evaluation questions are never available during memory construction. The sampled probe questions are used exclusively during training, as indicated in the RL Training Stage of Figure 1, and are drawn only from the training split. We use a split of 2:1:7 ratio (train, val and test), so no test conversation or question is seen during training. At test time, no probe question is provided: the attention features are obtained directly from the frozen LLM’s forward pass over the retained memory Mt−1M_{t-1} and the incoming window CtC_t. All test questions are answered only once, against the final memory MTM_T.

During training, a question enters the probe pool once the turns containing its evidence have arrived. The benchmark’s evidence positions are used only for this scheduling. They are never given to the policy, which does not observe which segments are evidence, and the reward depends solely on whether the frozen LLM’s answer matches the reference. To remove this dependence entirely, we also trained CURATE with a fixed generic probe (Appendix G.6), so the policy never sees a task question when computing its state. It still reaches 50.3 against 42.4 for Full Context while retaining 9.2% of the context (83.9% of gold vs. 2.3% of distractors kept). Training with question-specific probes therefore improves accuracy (53.6 vs. 50.3), but the policy learns to retain gold evidence and discard distractors even without access to any task question.

LoCoMo (n12)AccMem %Gold kept %Distractor kept %
Full Context42.4100100100
CURATE-CNN, generic probe only50.39.283.92.3
CURATE-CNN (paper)53.64.390.20.9
Oracle (no distractors)56.027.91000

W2 , Q2 (Contribution of RL-based selection):- As suggested, we compared CURATE with attention-only selection using the same retrieval heads, question formulation, and windowing, with no learned policy. Top-k ranking keeps the highest-scoring segments by retrieval-head attention, and threshold keeps segments whose attention exceeds a fixed value; both are set to CURATE’s memory budget. We also report the best threshold without the budget constraint.

LoCoMo (n12)AccMemG/D
Full Context0.4241.0001.00
Attention only, threshold0.1850.0436.33
Attention only, top-k ranking0.2830.04311.58
Attention only, best threshold (unmatched budget)0.4190.1175.19
CURATE0.5360.043100.2

Attention-only selection is itself a strong signal: with its best threshold, it reaches 0.419 while retaining only 11.7% of the context, close to Full Context at a fraction of the memory. This confirms that the retrieval heads carry useful relevance information. At a matched budget, however, CURATE clearly outperforms it (0.536 vs. 0.283), with a much higher G/D (100.2 vs. 11.58). The RL policy therefore makes better use of the same attention features, turning them into more reliable keep/discard decisions under a tight memory budget.

AuthorsOpenReview ↗

Part 2: Response (2/3)

Q3 (End-to-end Computational Efficiency) :- Yes, the latency in Table 5 includes the repeated trace generation and attention-extraction forward passes during memory construction. At every selection step, the frozen LLM generates a 16-token trace and runs one forward pass to read attention, and both are counted together with policy inference and final answering. We report the per-query and per-episode costs on LoCoMo, separately for feature extraction, policy inference, and final answer generation:

CURATE, end to endAvg per queryAvg per episode
LLM calls for selection (trace generation + attention pass)8.633.5
LLM calls for Answer generation13.9
Total processed prompt tokens (selection + answering)3,01311,749
Latency: feature extraction1.02 s3.97 s
Latency: policy inference0.01 s0.05 s
Latency: final answering0.08 s0.32 s
End-to-end latency1.11 s4.33 s

We re-measured the complete pipeline on a dedicated GPU with no other jobs running. Over three executions the end-to-end latency is 1.126 ± 0.01 s per query (1.11, 1.14, 1.13 s), consistent with and slightly below the 1.6 s in Table 5, where the timing is sensitive to GPU load. The breakdown is stable across runs: feature extraction accounts for about 92% of the time, final answering for 7%, and policy inference for 1%. The end-to-end latency remains an order of magnitude below the memory systems that call an LLM to write or consolidate memory at every turn. We will add this breakdown in the paper.

Q4 (Memory-R1 Comparison and Generalization):-

Memory-R1 manages memory with two RL-fine-tuned LLM agents: a Memory Manager that extracts facts from the dialogue and issues ADD / UPDATE / DELETE / NOOP operations on a memory bank, and an Answer Agent that filters the retrieved memories before answering. Both methods use outcome-driven RL, but they differ in what is trained and what is stored. Memory-R1 fine-tunes the full LLM (7-8B parameters) and stores rewritten facts; CURATE keeps the LLM frozen, trains only a 378K-parameter policy on the model's own attention, and makes the decision to keep or discard the segments.

We trained Memory-R1 using the authors’ code under the same distractor setting on LoCoMo (GRPO, Qwen2.5-7B).

MethodTrained parametersAccContext given to the LLM
Full Context–0.424100%
Memory-R17B (full LLM)0.43691%
CURATE (CNN)378K0.5364.3%

Memory-R1 performs on par with Full Context, while CURATE is 10 points higher with less than 5% of the context. Because the Memory Manager extracts facts from every turn, distractor turns are also stored and later retrieved for the reader. Memory-R1 learns how to organize memories, but it has no mechanism to recognize a stored statement as a distractor. The two approaches are therefore complementary: CURATE’s selection can be applied before a memory manager writes to its bank.

Retention on LongMemEval (50.1%) vs. LoCoMo (4.3%). The difference comes from the composition of the two benchmarks. On LoCoMo, distractors make up 74% of the segments, so most of the context can be removed; on LongMemEval they are only 12% of the segments (under 2% of the tokens), and the remaining 85% is genuine conversation, so even a perfect distractor filter retains 88% (Oracle). CURATE still retains much less than that (50.1%): beyond the distractors, it also learns to discard redundant content along with it.

AuthorsOpenReview ↗

Part 3: Response (3/3)

Q4 (Generalization across Distractor mechanisms) :- In practical RAG and agentic memory pipelines, distractors arise naturally: retrievers surface passages that are semantically similar to the query but misleading. Recent work shows that such distractors capture attention even at small proportions, so that even a small fraction of them sharply degrades long-context performance, and recovery requires removing them almost entirely (Gao et al., The First Drop of Ink, ICML 2026). Our retrieval-based distractors (2) reproduce this setting directly. To show generalization across different distractor mechanisms, we compared the trained CURATE policy with Full Context and AttentionRAG on two independently constructed distractor types, matching the distractor density. Both are built on the LoCoMo test conversations using the same pipeline as in the paper, with only the distractors replaced. In both cases, the LoCoMo-trained CNN policy is applied zero-shot.

(1) Conflicting distractors. The gold speaker restates the same fact with a conflicting value, so the context contains an explicit contradiction of the evidence. For example, the gold turn by John states, “we go a few times a week,” while the inserted distractor, also attributed to John, states, “I take my kids to the park every day.” Unlike the distractors in the paper, which attribute the competing detail to a third party, these directly contradict the gold evidence.

(2) Retrieval-based distractors. Instead of generating distractors, we retrieve them: for each question, a dense retriever returns the most similar real dialogue turns from other conversations, which are then inserted into the session. For example, for the gold turn by John, “We had to say goodbye to Max. He was such an important part of our family for 10 years,” the retriever returns a real turn by Maria from another conversation, “Max is my mother's cat, I took him when my mother passed away.” These mirror the hard negatives a RAG pipeline would surface; they are topically similar, naturally written, and not produced by any generator.

Distractor typeMethodAccMemG/D
ContradictoryFull Context0.3691.0001.00
ContradictoryRandom (matched budget)0.0960.0931.08
ContradictoryAttentionRAG0.3850.4781.31
ContradictoryCURATE (CNN)0.4730.09152.23
ContradictoryOracle (no distractors)0.5210.526–
Retrieval-basedFull Context0.4661.0001.00
Retrieval-basedRandom (matched budget)0.0630.0540.83
Retrieval-basedAttentionRAG0.4760.4741.23
Retrieval-basedCURATE (CNN)0.5230.05372.80
Retrieval-basedOracle (no distractors)0.5600.270–

Both mechanisms degrade Full Context (15.2 and 9.4 points below the Oracle). Applied zero-shot, CURATE improves over Full Context by 10.4 and 5.7 points while retaining only 9.1% and 5.3% of the context. AttentionRAG gains at most 1.6 points, keeps about 47% of the context, and retains distractors almost as often as gold evidence (G/D ≈ 1.3). CURATE therefore generalizes to distractor mechanisms it was not trained on, without requiring retraining. We will add these results to the revised paper.

Reference:- Gao et al. The First Drop of Ink: Nonlinear Impact of Misleading Information in Long-Context Reasoning. ICML 2026. arXiv:2605.10828.

Reviewer_JpFiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CURATE, which learns a policy that reads internal attention-level relevance signal directly from the executing model and combines it with an outcome-grounded RL objective to select the subset of an agent's accumulated context that should be forwarded to the next inference call. CURATE learns to identify and discard distractors without any relevance labels, retaining less than 10% of the context while preserving downstream accuracy.

Critical strengths

1, this paper is well motivated and address an important problem, how to improve the signal to noise ratio over a long multi-turn interactions with a learnt approach. It's useful that, can reduce the unnecessary context usage thus reduce the cost significantly, and it could also help with more concentrated signals for solving complicated problems that could have a lot of noise in the context window which could deviate the trajectory to a wrong path. 2, The experiments results show promising signals. CURATE improves over the strongest baseline by 8.4 percentage points on LoCoMo and 4.0 points on HotpotQA, while retaining 4.3% and 6.0% of the context, respectively.

Critical weaknesses
  1. It is unclear where the probe questions come from at test time. Does the selector see the eventual evaluation questions while building memory? This matters for whether the method can remember useful information before future questions are known.

  2. The latency comparison needs more details. Selecting context requires extra LLM computation, but it is unclear whether the reported time includes this kind of effort.

  3. The experiments use a narrow type of generated distractor. The transfer experiment uses similar distractors, so robustness to other kinds of misleading context remains unclear, will be better to have some measurements.

Questions for authors

How are probe questions obtained at test time? Can the selector access the eventual evaluation questions while building memory? How does it perform when those questions are withheld?

Does the latency in Table 5 include probe generation, attention extraction, memory construction, and final answering?

4 comments
AuthorsOpenReview ↗

Part 1: Response (1/2)

We sincerely thank the reviewer for their thoughtful feedback and constructive evaluation of our work.

W1 , Q1 (Probe questions at test time) :- No, the selector never sees the eventual evaluation questions while building memory. The sampled probe questions are used exclusively during training, as indicated in the RL Training Stage of Figure 1, and are drawn only from the training split. We use a split of 2:1:7 ratio (train, val and test), so no test conversation or question is seen during training. At test time, no probe question is provided: the attention features are obtained directly from the frozen LLM’s forward pass over the retained memory Mt−1M_{t-1} and the incoming window CtC_t. All test questions are answered only once, against the final memory MTM_T.

To rule out any benefit from question-specific probes, we also trained a policy using only the generic probe of Appendix G.6, so it never sees a task question. It still reaches 50.3 against 42.4 for Full Context while retaining 9.2% of the context (83.9% of gold vs. 2.3% of distractors kept). Training with question-specific probes therefore improves accuracy (53.6 vs. 50.3), but the policy learns to retain gold evidence and discard distractors even without access to any task question.

LoCoMo (n12)AccMem %Gold kept %Distractor kept %
Full Context42.4100100100
CURATE-CNN, generic probe only50.39.283.92.3
CURATE-CNN (paper)53.64.390.20.9
Oracle (no distractors)56.027.91000

W2 , Q2 (Latency breakdown) :- Yes, the latency in Table 5 includes probe generation, attention extraction, memory construction, and final answering. At every selection step, the frozen LLM generates a 16-token trace and runs one forward pass to read attention, and both are counted together with policy inference and final answering. We report a detailed breakdown on LoCoMo:

CURATE, end to endAvg per queryAvg per episode
LLM calls for selection (trace generation + attention pass)8.633.5
LLM calls for Answer generation13.9
Total processed prompt tokens (selection + answering)3,01311,749
Latency: feature extraction1.02 s3.97 s
Latency: policy inference0.01 s0.05 s
Latency: final answering0.08 s0.32 s
End-to-end latency1.11 s4.33 s

We re-measured the complete pipeline on a dedicated GPU with no other jobs running. Over three executions the end-to-end latency is 1.126 ± 0.01 s per query (1.11, 1.14, 1.13 s), consistent with and slightly below the 1.6 s in Table 5, where the timing is sensitive to GPU load. The breakdown is stable across runs: feature extraction accounts for about 92% of the time, final answering for 7%, and policy inference for 1%. We will add this breakdown in the paper.

AuthorsOpenReview ↗

Part 2: Response (2/2)

W3 (Generalization across Distractor mechanisms) :- We thank the reviewer for this suggestion. To show generalization across different distractor mechanisms, we compared the trained CURATE policy with Full Context and AttentionRAG on two independently constructed distractor types, matching the distractor density. Both are built on the LoCoMo test conversations using the same pipeline as in the paper, with only the distractors replaced. In both cases, the LoCoMo-trained CNN policy is applied zero-shot.

(1) Conflicting distractors. The gold speaker restates the same fact with a conflicting value, so the context contains an explicit contradiction of the evidence. For example, the gold turn by John states, “we go a few times a week,” while the inserted distractor, also attributed to John, states, “I take my kids to the park every day.” Unlike the distractors in the paper, which attribute the competing detail to a third party, these directly contradict the gold evidence.

(2) Retrieval-based distractors. Instead of generating distractors, we retrieve them: for each question, a dense retriever returns the most similar real dialogue turns from other conversations, which are then inserted into the session. For example, for the gold turn by John, “We had to say goodbye to Max. He was such an important part of our family for 10 years,” the retriever returns a real turn by Maria from another conversation, “Max is my mother's cat, I took him when my mother passed away.” These mirror the hard negatives a RAG pipeline would surface; they are topically similar, naturally written, and not produced by any generator.

Distractor typeMethodAccMemG/D
ContradictoryFull Context0.3691.0001.00
ContradictoryRandom (matched budget)0.0960.0931.08
ContradictoryAttentionRAG0.3850.4781.31
ContradictoryCURATE (CNN)0.4730.09152.23
ContradictoryOracle (no distractors)0.5210.526–
Retrieval-basedFull Context0.4661.0001.00
Retrieval-basedRandom (matched budget)0.0630.0540.83
Retrieval-basedAttentionRAG0.4760.4741.23
Retrieval-basedCURATE (CNN)0.5230.05372.80
Retrieval-basedOracle (no distractors)0.5600.270–

Both mechanisms degrade Full Context (15.2 and 9.4 points below the Oracle). Applied zero-shot, CURATE improves over Full Context by 10.4 and 5.7 points while retaining only 9.1% and 5.3% of the context. AttentionRAG gains at most 1.6 points, keeps about 47% of the context, and retains distractors almost as often as gold evidence (G/D ≈ 1.3). CURATE therefore generalizes to distractor mechanisms it was not trained on, without requiring retraining. We will add these results to the revised paper.

AuthorsOpenReview ↗

A Gentle Reminder to Reviewer JpFi

Dear Reviewer,

Thank you for your valuable comments, which we have addressed in detail in our response. We would be happy to clarify any further concerns. If our responses resolve your concerns, we would be grateful if you would consider reassessing your score.

Best wishes, Authors

AuthorsOpenReview ↗

Reminder to check our responses

Dear Reviewer JpFi,

Thank you for your valuable comments, which we have addressed in detail in our response. As the discussion phase is concluding soon, we would greatly appreciate it if you could kindly review our responses and consider reassessing our submission.

Thank you once again for your time and consideration.

Reviewer_WsvAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CURATE, an RL-based context selection policy using attention signals from a frozen language model. Experiments on distractor-augmented LoCoMo and HotpotQA show improved question-answering accuracy while retaining only 4.3% and 6.0% of the original context, respectively.

Critical strengths
  1. The paper addresses context management in long-horizon LLM agents, particularly the challenge of distinguishing useful evidence from semantically related distractors. This is a meaningful problem, and the motivation for exploiting the model's internal relevance signals is compelling.
  2. CURATE combines attention-based features with a small RL-trained selection policy while keeping the underlying LLM frozen. The distinction between pooled and unpooled attention representations is interesting, and the reported advantage of the CNN policy suggests that token-level attention patterns may contain useful information beyond aggregate attention scores.
Critical weaknesses
  1. CURATE relies on sampled probe questions to extract attention features during context selection (Section 4.1 and Appendix G.6). However, it is unclear how these questions are obtained at inference time, when future user queries are generally unknown. If evaluation questions are used during memory construction, this may introduce privileged information and limit the validity of the online memory setting. The authors should clarify the inference protocol and evaluate the method without access to future queries.
  2. The attention analysis already shows strong separation between gold evidence and distractors, raising the question of whether a simple attention-based selection strategy could achieve similar results. The current CNN-versus-MLP comparison does not isolate the contribution of RL. Comparisons with attention-score top-k selection, threshold-based filtering, and random-head policies under matched memory budgets would strengthen the methodological claims.
  3. The evaluation primarily relies on artificially injected distractors that describe third-party entities (Appendix G). Such distractors may exhibit systematic patterns that make them easier to distinguish from gold evidence. Moreover, the LongMemEval generalization experiment also introduces synthetic distractors. Evaluating on natural retrieval noise, unmodified benchmarks, and more diverse distractor types would better establish robustness in realistic settings.
  4. The baselines operate under substantially different memory budgets, making it difficult to attribute performance improvements solely to better context selection. In addition, MemAgent and MEM1 are reported as retaining 100% of memory despite using compressed representations, suggesting that memory accounting may not be directly comparable. The authors should clarify the memory metric and provide accuracy-memory comparisons under matched budgets.
  5. Although CURATE uses a lightweight policy, attention feature extraction requires LLM generation and an additional forward pass at each selection step. The reported per-query latency (Appendix D) does not clearly establish whether the full cost of incremental memory maintenance is included. End-to-end episode latency, total processed tokens, and the number of LLM forward passes would provide a more convincing efficiency comparison.
  6. The full policy evaluation is conducted only with Qwen2.5-7B, while Llama-3.1-8B is used only for attention analysis. Furthermore, the main results lack confidence intervals or multiple-seed evaluations, making it difficult to assess the reliability of relatively small performance differences. Additional backbone evaluations, multiple training seeds, and uncertainty estimates would strengthen the conclusions.
Questions for authors

See weakness part

8 comments
AuthorsOpenReview ↗

Part 1: Response (1/3)

We sincerely thank the reviewer for their thoughtful feedback and constructive evaluation of our work.

W1 (Inference protocol and access to future queries) :- No future query is available during memory construction. The sampled probe questions are used exclusively during training, as indicated in the RL Training Stage of Figure 1, and are drawn only from the training split. We use a split of 2:1:7 ratio (train, val and test), so no test conversation or question is seen during training. At inference time, the protocol is fully online: no probe question is provided, and the attention features are obtained directly from the frozen LLM’s forward pass over the retained memory Mt−1M_{t-1} and the incoming window CtC_t. The evaluation questions are revealed only after the final window and answered once, against the final memory MTM_T. All reported results follow this protocol.

To further evaluate the method without access to any task question, as suggested, we also trained a policy using only the generic probe of Appendix G.6, so it never sees a task question. It still reaches 50.3 against 42.4 for Full Context while retaining 9.2% of the context (83.9% of gold vs. 2.3% of distractors kept). Question-specific probes during training therefore improve accuracy (53.6 vs. 50.3), but they are not required: the policy learns to retain gold evidence and discard distractors without access to any task question, consistent with the online memory setting.

LoCoMo (n12)AccMem %Gold kept %Distractor kept %
Full Context42.4100100100
CURATE-CNN, generic probe only50.39.283.92.3
CURATE-CNN (paper)53.64.390.20.9
Oracle (no distractors)56.027.91000

W2 (Contribution of RL-based selection):- As suggested, we compared CURATE with attention-only selection using the same retrieval heads, question formulation, and windowing, with no learned policy. Top-k ranking keeps the highest-scoring segments by retrieval-head attention, and threshold keeps segments whose attention exceeds a fixed value; both are set to CURATE’s memory budget. We also report top-k ranking with randomly selected heads, and the best threshold without the budget constraint. Since the features are identical and only the decision rule differs, this comparison isolates the contribution of RL.

LoCoMo (n12)AccMemG/D
Full Context0.4241.0001.00
Random segments0.0830.0480.82
Attention only, top-k ranking, random heads0.1390.0434.19
Attention only, threshold0.1850.0436.33
Attention only, top-k ranking0.2830.04311.58
Attention only, best threshold (unmatched budget)0.4190.1175.19
CURATE0.5360.043100.2

Random selection at the same budget collapses to 0.083, so the budget alone does not explain the gains. Attention-only selection is itself a strong signal, consistent with the separation observed in our attention analysis: with its best threshold, it reaches 0.419 while retaining only 11.7% of the context, close to Full Context at a fraction of the memory. Replacing the retrieval heads with random heads halves top-k accuracy (0.139 vs. 0.283), confirming that the relevance signal comes specifically from the retrieval heads. At a matched budget, however, CURATE clearly outperforms attention-only selection (0.536 vs. 0.283), with a much higher G/D (100.2 vs. 11.58). The RL policy therefore makes better use of the same attention features, turning them into more reliable keep/discard decisions under a tight memory budget.

AuthorsOpenReview ↗

Part 2: Response (2/3)

W3 (Generalization to diverse distractor types):-

Unmodified benchmark. Table 3 already reports every method on LoCoMo with and without the injected distractors. On the unmodified benchmark, CURATE (CNN) reaches 0.554, matching MemoryOS (0.554) and within 0.006 of the best baseline, MemGPT (0.560), while retaining only a small fraction of the context. CURATE is also the most robust method: its accuracy drops by only 0.006 (MLP) and 0.018 (CNN) when distractors are added, compared against large margin drops for other baselines. CURATE therefore remains competitive when no distractors are present, and it degrades the least when they are introduced.

We further evaluated CURATE on two additional distractor types. We compared the trained CURATE policy with Full Context and AttentionRAG on two independently constructed distractor types, matching the distractor density. In both cases, the CNN policy that was trained on original distractor setting is applied zero-shot.

(1) Conflicting distractors. The gold speaker restates the same fact with a conflicting value, so the context contains an explicit contradiction of the evidence. For example, the gold turn by John states, “we go a few times a week,” while the inserted distractor, also attributed to John, states, “I take my kids to the park every day.” Unlike the distractors in the paper, which attribute the competing detail to a third party, these directly contradict the gold evidence.

(2) Retrieval-based distractors. Instead of generating distractors, we retrieve them: for each question, a dense retriever returns the most similar real dialogue turns from other conversations, which are then inserted into the session. For example, for the gold turn by John, “We had to say goodbye to Max. He was such an important part of our family for 10 years,” the retriever returns a real turn by Maria from another conversation, “Max is my mother's cat, I took him when my mother passed away.” These mirror the hard negatives a RAG pipeline would surface; they are topically similar, naturally written, and not produced by any generator.

Distractor typeMethodAccMemG/D
ContradictoryFull Context0.3691.0001.00
ContradictoryRandom0.0960.0931.08
ContradictoryAttentionRAG0.3850.4781.31
ContradictoryCURATE (CNN)0.4730.09152.23
ContradictoryOracle (no distractors)0.5210.526–
Retrieval-basedFull Context0.4661.0001.00
Retrieval-basedRandom0.0630.0540.83
Retrieval-basedAttentionRAG0.4760.4741.23
Retrieval-basedCURATE (CNN)0.5230.05372.80
Retrieval-basedOracle (no distractors)0.5600.270–

Both mechanisms degrade Full Context (15.2 and 9.4 points below the Oracle). Applied zero-shot, CURATE improves over Full Context by 10.4 and 5.7 points while retaining only 9.1% and 5.3% of the context. AttentionRAG gains at most 1.6 points, keeps about 47% of the context, and retains distractors almost as often as gold evidence (G/D ≈ 1.3). CURATE therefore generalizes to distractor mechanisms it was not trained on, without requiring retraining.

W4 (Matched-budget comparison):- We agree that accuracy should be compared at equal memory. We ran every method whose budget can be set at CURATE’s budget. The table also includes the baselines from Table 2 whose memory already falls in the same range (at most 7%).

Method (LoCoMo)AccMem
Random0.0830.048
RAG, dense (k=3)0.1230.041
RAG, TF-IDF (k=3)0.1830.041
Attention top-k (retrieval heads)0.2830.043
A-Mem0.2810.068
Mem00.3060.062
MemAgent0.4520.061
CURATE (CNN)0.5360.043

At the same budget, CURATE is the most accurate method: 8.4 points above the strongest baseline (MemAgent) and 23 to 45 points above the others. The remaining baselines (AttentionRAG, MemoryOS, MemGPT, MEM1) cannot be placed at this budget because they determine their own memory size, either through an LLM that writes, consolidates, or pages its memory, or through their own pruning rule. They use far more memory than CURATE (41% to 100%) and still score lower (0.316 to 0.427). For baselines whose budget can be varied, accuracy increases with memory: dense RAG rises from 0.123 at 4.1% to 0.418 as its budget grows. The larger budgets in Table 2 therefore favor the baselines, and CURATE outperforms all of them while using small budget.

Memory accounting for MemAgent and MEM1. You are right about MemAgent, and reporting it at 100 % was a mistake on our part. It reads the context in chunks, writes its own memory, and answers from that memory alone, so its actual memory is 6.1%. For MEM1, 100 % is accurate in our evaluation, since it receives the full context in a single prompt and answers directly from it. We will correct this value in the revised table.

AuthorsOpenReview ↗

Part 3: Response (3/3)

W5 (End-to-end cost) :- Yes, the latency in Appendix D includes the full cost of incremental memory maintenance. At every selection step, the frozen LLM generates a 16-token trace and runs one forward pass to read attention, and both are counted together with policy inference and final answering. We report the three requested quantities on LoCoMo:

CURATE, end to endAvg per queryAvg per episode
LLM calls for selection (trace generation + attention pass)8.633.5
LLM calls for Answer generation13.9
Total processed prompt tokens (selection + answering)3,01311,749
Latency: feature extraction1.02 s3.97 s
Latency: policy inference0.01 s0.05 s
Latency: final answering0.08 s0.32 s
End-to-end latency1.11 s4.33 s

We re-measured the complete pipeline on a dedicated GPU with no other jobs running. Over three executions the end-to-end latency is 1.126 ± 0.01 s per query (1.11, 1.14, 1.13 s), consistent with and slightly below the 1.6 s in Table 5, where the timing is sensitive to GPU load. The breakdown is stable across runs: feature extraction accounts for about 92% of the time, final answering for 7%, and policy inference for 1%. The end-to-end latency remains an order of magnitude below the memory systems that call an LLM to write or consolidate memory at every turn. We will add this breakdown in the paper.

W6 (Second backbone and uncertainty estimates):- We trained the full CURATE policy on Llama-3.1-8B-Instruct, using the retrieval heads identified for that model and the same configuration as for Qwen2.5-7B:

Method (Llama-3.1-8B-Instruct, LoCoMo, n12)AccMemG/D
Full Context0.4661.0001.00
Random (matched budget)0.0640.0680.78
RAG (k=3)0.1790.0413.96
RAG (k=10)0.3990.1352.35
RAG (k=20)0.3970.2711.69
RAG (k=50)0.4300.6631.24
RAG (k=100)0.4550.9711.03
AttentionRAG0.5010.8791.02
A-Mem0.2360.0672.94
Mem00.2940.0583.79
MemoryOS0.2580.4111.28
MemGPT0.3490.6461.19
CURATE (CNN)0.5240.069109.7
Oracle (no distractors)0.5510.270–

With Llama-3.1-8B, CURATE reaches 0.524 against 0.466 for Full Context while retaining 6.9% of the context. This behavior matches that on Qwen2.5-7B (0.536 vs. 0.424 at 4.3%), so the method is not specific to one backbone and generalizes across different model families.

To quantify uncertainty, we computed paired bootstrap confidence intervals over the evaluation episodes (10,000 resamples):

Acc95% CI
CURATE (CNN, Qwen)0.536[0.433, 0.639]
Full Context0.424[0.329, 0.522]
CURATE − Full Context+0.112[+0.025, +0.210]

The improvement over Full Context is positive in 99.6% of resamples, and the 95% confidence interval of the difference excludes zero.

We further trained CURATE with three random seeds under the same settings and the Qwen2.5-7B model as in main Table 2. Across seeds, it reaches 0.524 ± 0.016 (mean ± s.d.) while retaining 5.4% of the context on average, consistently above Full Context (0.424). In the revised paper, we will extend the seed analysis to the other baselines.

CURATE (CNN), 3 seedsAccMem
Mean ± s.d.0.524 ± 0.0160.054

We again thank the reviewer for these valuable questions, and we will add these results and experiments to the revised paper.

Reviewer_6pjYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CURATE trains a lightweight memory-selection policy using an LLM’s own attention patterns and rewards for accurate answers with compact context. It learns to discard misleading and unnecessary information without explicit relevance labels, improving accuracy while retaining just 4–6% of context on two tested benchmarks. The results support attention-guided memory selection, although broader generalization and applicability to closed models remain limited.

Critical strengths
  1. Task-aligned memory selection. CURATE combines internal attention signals with answer-level rewards to learn context selection without relevance labels, while keeping the LLM frozen.

  2. Accuracy and compression improve together. CURATE improves accuracy by 11.2 and 9.0 percentage points on LoCoMo and HotpotQA while retaining only 4.3% and 6.0% of context, demonstrating substantial gains in the tested settings.

Critical weaknesses
  1. Incomplete specification of inference-time probing.: Section 4.1 (p. 4, paragraph beginning “At step (t)”) conditions selection on query (q), while Appendix G.6 (p. 20, “Feature extraction”) describes sampled probes during training without clarifying their source at inference. Please specify whether eventual evaluation questions are available during memory construction, as this affects whether the results support memory retention for unknown future queries.

  2. The contribution of long-horizon credit assignment is not isolated. The existing ablations do not establish whether the gains require optimizing future outcomes. Alternatively, we can also try against a myopic policy using identical attention features and architecture but optimizing only the current step’s reward, with comparable context-retention levels.

  3. Generalization across distractor mechanisms remains untested. The LongMemEval experiment uses the same synthetic distractor pipeline, limiting what it establishes about robustness beyond that construction. Perhaps it is interesting to compare the trained CURATE policy with Full Context and AttentionRAG on independently constructed distractors, preserving gold evidence and matching distractor density.

Questions for authors

See the critic weaknesses above.

4 comments
AuthorsOpenReview ↗

Part 1: Response (1/2)

We sincerely thank the reviewer for their thoughtful feedback and constructive evaluation of our work.

W1 (Inference-time probing) :- Evaluation questions are never available during memory construction. The sampled probe questions are used exclusively during training, as indicated in the RL Training Stage of Figure 1, and are drawn only from the training split. We use a split of 2:1:7 ratio (train, val and test), so no test conversation or question is seen during training. At test time, no probe question is provided: the attention features are obtained directly from the frozen LLM’s forward pass over the retained memory Mt−1M_{t-1} and the incoming window CtC_t. All test questions are answered only once, against the final memory MTM_T.

To rule out any benefit from question-specific probes, we also trained a policy using only the generic probe of Appendix G.6, so it never sees a task question. It still reaches 50.3 against 42.4 for Full Context while retaining 9.2% of the context (83.9% of gold vs. 2.3% of distractors kept). Training with question-specific probes therefore improves accuracy (53.6 vs. 50.3), but the policy learns to retain gold evidence and discard distractors even without access to any task question.

LoCoMo (n12)AccMem %Gold kept %Distractor kept %
Full Context42.4100100100
CURATE-CNN, generic probe only50.39.283.92.3
CURATE-CNN (paper)53.64.390.20.9
Oracle (no distractors)56.027.91000

We will add this clarification and the result in the revised paper.

W2 (Contribution of long-Horizon credit assignment) :- As suggested, we trained a myopic policy with identical attention features, architecture, curriculum, and hyperparameters, where each decision is credited only with its own step reward and the global reward is removed. The table reports validation accuracy and memory at each curriculum stage (n3, n7, n12 = 3, 7, 12 distractors per question).

Myopic (step only)CURATE (step + global)
StageAccMemAccMem
n30.5170.0760.4660.234
n70.4720.0620.4550.217
n120.1780.0080.5040.160

When distractors are sparse (n3, n7), the myopic policy matches or exceeds CURATE at lower retention. At n12, however, it collapses: retention falls below 1%, gold evidence is discarded along with distractors, and accuracy drops to 0.178 without recovering. CURATE remains stable and reaches its best accuracy at n12 (0.504). The step reward is thus an effective local signal, but under dense distractors it drives a myopic policy toward discarding everything. The global reward, which scores the final memory against all questions of the episode, credits evidence kept for later questions and keeps the policy stable.

To further show the importance of long-horizon credit assignment, We compared CURATE against attention-only selection (same retrieval heads and windowing, no learned policy, set to CURATE’s memory budget) and against the same policy trained with the global reward only:

LoCoMo (n12)AccMemG/D
Full Context0.4241.0001.00
Attention only, threshold0.1850.0436.33
Attention only, best threshold (unmatched budget)0.4190.1175.19
Attention only, top-k ranking0.2830.04311.58
RL policy, global reward only0.4820.04592.5
CURATE (step + global reward)0.5360.043100.2

Attention alone reaches only 0.283 at the same budget, and even with its threshold tuned freely it peaks at 0.419 while retaining 11.7% of the context, still below Full Context. The learned policy is therefore essential. With the global reward alone, the policy recovers most of the gain (0.482). With the step reward alone, it collapses at n12, as shown above. Combining the two gives the full result (0.536). Both terms are therefore needed: the step reward for efficient learning, and the global reward for long-horizon credit and stability under dense distractors. We will add both analyses to the revised paper.

AuthorsOpenReview ↗

Part 2: Response (2/2)

W3 (Generalization across Distractor mechanisms) :- We thank the reviewer for this suggestion. To show generalization across different distractor mechanisms, we compared the trained CURATE policy with Full Context and AttentionRAG on two independently constructed distractor types, matching the distractor density. Both are built on the LoCoMo test conversations using the same pipeline as in the paper, with only the distractors replaced. In both cases, the LoCoMo-trained CNN policy is applied zero-shot.

(1) Conflicting distractors. The gold speaker restates the same fact with a conflicting value, so the context contains an explicit contradiction of the evidence. For example, the gold turn by John states, “we go a few times a week,” while the inserted distractor, also attributed to John, states, “I take my kids to the park every day.” Unlike the distractors in the paper, which attribute the competing detail to a third party, these directly contradict the gold evidence.

(2) Retrieval-based distractors. Instead of generating distractors, we retrieve them: for each question, a dense retriever returns the most similar real dialogue turns from other conversations, which are then inserted into the session. For example, for the gold turn by John, “We had to say goodbye to Max. He was such an important part of our family for 10 years,” the retriever returns a real turn by Maria from another conversation, “Max is my mother's cat, I took him when my mother passed away.” These mirror the hard negatives a RAG pipeline would surface; they are topically similar, naturally written, and not produced by any generator.

Distractor typeMethodAccMemG/D
ContradictoryFull Context0.3691.0001.00
ContradictoryRandom (matched budget)0.0960.0931.08
ContradictoryAttentionRAG0.3850.4781.31
ContradictoryCURATE (CNN)0.4730.09152.23
ContradictoryOracle (no distractors)0.5210.526–
Retrieval-basedFull Context0.4661.0001.00
Retrieval-basedRandom (matched budget)0.0630.0540.83
Retrieval-basedAttentionRAG0.4760.4741.23
Retrieval-basedCURATE (CNN)0.5230.05372.80
Retrieval-basedOracle (no distractors)0.5600.270–

Both mechanisms degrade Full Context (15.2 and 9.4 points below the Oracle). Applied zero-shot, CURATE improves over Full Context by 10.4 and 5.7 points while retaining only 9.1% and 5.3% of the context. AttentionRAG gains at most 1.6 points, keeps about 47% of the context, and retains distractors almost as often as gold evidence (G/D ≈ 1.3). CURATE therefore generalizes to distractor mechanisms it was not trained on, without requiring retraining. We will add these results to the revised paper.

AuthorsOpenReview ↗

A Gentle Reminder to Reviewer 6pjY

Dear Reviewer,

Thank you for your valuable comments, which we have addressed in detail in our response. We would be happy to clarify any further concerns. If our responses resolve your concerns, we would be grateful if you would consider reassessing your score.

Best wishes, Authors

AuthorsOpenReview ↗

Reminder to check our responses

Dear Reviewer 6pjY,

Thank you for your valuable comments, which we have addressed in detail in our response. As the discussion phase is concluding soon, we would greatly appreciate it if you could kindly review our responses and consider reassessing our submission.

Thank you once again for your time and consideration.

Reviewer_3c4kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CURATE, a policy that selects which parts of an agent’s interaction history to retain. A dialogue example illustrates the problem. When asked about the speaker’s relationship status, a model answers “engaged” after reading about a relative’s wedding plans, even though the speaker’s own statement that she is single remains in the conversation. The authors develop a pipeline that generates similarly plausible distractors and checks them for contradictions with the original facts.

CURATE keeps the answering model frozen and uses held-out examples to identify the 10% of attention heads that attend most strongly to the context. Across Qwen and Llama, these heads assign the most attention on average to supporting evidence, followed by distractors and then redundant material. A lightweight policy uses these attention patterns to decide which segments enter memory. Reinforcement learning rewards correct answers from the retained memory and penalizes its size, allowing the policy to learn without segment-level relevance labels.

On LoCoMo with 12 distractors per question, CURATE’s convolutional selector raises Qwen2.5-7B-Instruct’s accuracy from 42.4% to 53.6% while retaining 4.3% of the context. On modified HotpotQA, it reaches 68.0% accuracy with 6.0% retained, compared with 66.0% when only the injected distractors are removed. Ablations favor preserving token-level attention patterns and gradually increasing distractor density during training. On LongMemEval with added distractors, the LoCoMo-trained policy improves accuracy from 50.8% to 58.5% without retraining, though it retains 50.1% of the context on this task.

Critical strengths

The paper designs its distractors around a specific retrieval error: a passage can contain a plausible answer while describing the wrong person or event. The authors check that each added passage is consistent with the original facts, so detecting a contradiction alone cannot solve the task. Table 1 then connects this challenge to the proposed method. Across four datasets and two model families, attention heads selected on held-out HotpotQA examples give supporting evidence more weight on average than distractors, which in turn receive more weight than unrelated material. This ordering gives a concrete reason to use the answering model’s attention as input to a memory policy trained without segment-level relevance labels.

With the comparison with random retention, on LoCoMo with 12 distractors per question, CURATE reaches 53.6% accuracy while retaining 4.3% of the context. Random selection retains a slightly larger share, 4.8%, yet reaches only 8.3% accuracy. Supplying the full history yields 42.4% (Table 2). The random baseline matters because it shows how much the choice of passages affects answer quality at a comparable memory budget. The benefit also extends to the modified HotpotQA task, where CURATE raises accuracy from 59.0% to 68.0% while retaining 6.0% of the context.

Critical weaknesses

My main concern is what CURATE knows when it decides to discard a passage. Appendix G.6 explains that, during training, the model answers a sampled probe question before the selector makes its decision. The paper leaves unclear how probes are chosen at test time and whether they can overlap with the questions used to evaluate the final memory. In the dialogue example, a probe about relationship status could direct attention to the relevant sentence while the selector still has access to it. That distinction matters for an agent expected to remember information for requests it has yet to receive. I would give the results more weight if the authors built memory using only probes derived from the conversation observed so far, then evaluated it on questions withheld until memory construction was complete.

My second concern is how much sequential reinforcement learning contributes beyond the attention features themselves. Table 1 shows that the selected heads already assign higher average scores to evidence than to distractors. The random baseline establishes that the choice of passages matters, but the closest attention-based comparison operates at a substantially different budget. On LoCoMo, AttentionRAG retains 43.5% of the context, versus 4.3% for CURATE. A simple threshold on scores from the same heads and probes, calibrated to match CURATE’s retention budget, would show how much the learned selector adds. Comparing the same network trained with immediate rewards alone would further clarify whether assigning credit across the full episode improves selection.

Questions for authors

Clarification requests

  1. Appendix G.6 describes how probes are sampled during training. At test time, how is each probe chosen before CURATE decides which segments to retain? Can it come from the questions later used to score the final memory, or does its selection depend on annotations identifying which questions are already answerable? Could you walk through one test episode, showing the probe available at selection time and when the final evaluation question is first revealed?

  2. On LoCoMo, Table 2 reports that AttentionRAG retains 43.5% of the context and CURATE retains 4.3%. Have you evaluated a simple rule that scores segments using CURATE’s own calibrated heads and probes, with a threshold set on validation data to match its retention budget? If so, what accuracy and retained-context fraction did that rule achieve on LoCoMo and HotpotQA?

Question for a follow-up experiment

  1. With the CNN architecture, probe access, and memory budget held fixed, how much does accuracy change when the selector learns from immediate rewards alone versus receiving credit for answers later in the episode? This comparison would clarify how much the long-horizon training objective contributes.
4 comments
AuthorsOpenReview ↗

Part 1: Response (1/2)

We sincerely thank the reviewer for their thoughtful feedback and constructive evaluation of our work.

W1 , Q1 (Probe Question):- Evaluation questions are never available during memory construction. The sampled probe questions are used exclusively during training, as indicated in the RL Training Stage of Figure 1, and are drawn only from the training split. We use a split of 2:1:7 ratio (train, val and test), so no test conversation or question is seen during training. At test time, no probe question is provided: the attention features are obtained directly from the frozen LLM’s forward pass over the retained memory Mt−1M_{t-1} and the incoming window CtC_t. All test questions are answered only once, against the final memory MTM_T.

During training, a question enters the probe pool once the turns containing its evidence have arrived. The benchmark’s evidence positions are used only for this scheduling. They are never given to the policy, which does not observe which segments are evidence, and the reward depends solely on whether the frozen LLM’s answer matches the reference. To remove this dependence entirely, we also trained CURATE with a fixed generic probe (Appendix G.6), so the policy never sees a task question when computing its state. It still reaches 50.3 against 42.4 for Full Context while retaining 9.2% of the context (83.9% of gold vs. 2.3% of distractors kept). Training with question-specific probes therefore improves accuracy (53.6 vs. 50.3), but the policy learns to retain gold evidence and discard distractors even without access to any task question.

LoCoMo (n12)AccMem %Gold kept %Distractor kept %
Full Context42.4100100100
CURATE-CNN, generic probe only50.39.283.92.3
CURATE-CNN (paper)53.64.390.20.9
Oracle (no distractors)56.027.91000

Walkthrough of one episode. To further clarify what CURATE observes and when, we walk through one training episode and one test episode below.

Training (one episode from the training split):

  • The conversation arrives in windows of segments, and the memory starts empty.
  • At each step, the frozen LLM reads the current memory and the new window together with a probe, which is either a sampled training question or the generic probe described above. Its retrieval-head attention is passed to the policy.
  • The policy keeps or discards each new segment.
  • Step reward: training questions whose evidence has already appeared are answered from the current memory, minus a cost for the memory kept.
  • Episode reward: after the last window, all questions of the episode are answered from the final memory.
  • The policy itself observes only the attention features. It never sees the questions, answers, or evidence annotations, which are used only to construct probes and compute rewards.

Test (one held-out episode):

  • At each step, the frozen LLM reads only the current memory and the new window, and the policy keeps or discards each new segment.
  • No question, answer, or evidence annotation is used at any step.
  • The evaluation questions are revealed for the first time after the final window, once the memory is fixed.
  • Each question is answered from the final memory alone, and this is the accuracy we report.

W2 , Q2 (Attention-only selection at a matched budget):-

As suggested, we compared CURATE with attention-only selection that uses the same retrieval heads, question formulation, and windowing, with no learned policy. Threshold keeps segments whose attention exceeds a value calibrated to match CURATE’s retention budget, and top-k ranking keeps the highest-scoring segments at the same budget. We also report the best unmatched threshold result.

MethodLoCoMo AccLoCoMo MemHotpotQA AccHotpotQA Mem
Full Context0.4241.0000.5901.000
Random segments0.0830.0480.2130.063
Attention only, threshold0.1850.0430.6000.050
Attention only, top-k0.2830.0430.5500.050
Attention only, best threshold (unmatched budget)0.4190.1170.6500.108
CURATE0.5360.0430.6800.060

Random selection at the same budget collapses on both benchmarks (0.083 and 0.213), so the budget alone does not explain the gains. Attention-only selection is itself a strong signal. With its best threshold, it reaches 0.419 on LoCoMo at 11.7% of the context and 0.650 on HotpotQA at 10.8%, and on HotpotQA it already exceeds Full Context at a matched budget (0.600). At a matched budget, however, CURATE outperforms it on both benchmarks (0.536 vs. 0.283 on LoCoMo and 0.680 vs. 0.600 on HotpotQA), and it also beats the best unconstrained threshold on both with less memory (4.3% vs. 11.7% and 6.0% vs. 10.8%). The RL policy therefore makes better use of the same attention features, turning them into more reliable keep/discard decisions under a tight memory budget.

AuthorsOpenReview ↗

Part 2: Response (2/2)

Q3 Follow up Experiment (Contribution of long-Horizon credit assignment) :- As suggested, we trained the same CNN policy with immediate (step) rewards only, keeping the attention features, probe access, curriculum, and hyperparameters identical, and removing the global reward. The table reports validation accuracy and memory at each curriculum stage (n3, n7, n12 = 3, 7, 12 distractors per question).

Myopic (step reward only)‖CURATE (step + global reward)
StageAccMem‖AccMem
n30.5170.076‖0.4660.234
n70.4720.062‖0.4550.217
n120.1780.008‖0.5040.160

When distractors are sparse (n3, n7), the immediate-reward policy matches or exceeds CURATE at lower retention. At n12, however, it collapses: retention falls below 1%, gold evidence is discarded along with distractors, and accuracy drops to 0.178 without recovering. The memory budget therefore cannot be held fixed for this policy at high density, since the collapse of retention is itself the failure mode. CURATE remains stable and reaches its best accuracy at n12 (0.504). The immediate reward is an effective local signal, but under dense distractors it drives the policy toward discarding everything. In addition, training with the global reward alone reaches 0.482 on the test split, and combining both rewards gives 0.536. Both terms are therefore needed: the immediate reward for efficient learning, and the long-horizon reward for stability under dense distractors.

We will add the clarification regarding the probe question and results to the revised paper. We thank the reviewer for the constructive feedback.

AuthorsOpenReview ↗

A Gentle Reminder to Reviewer 3c4k

Dear Reviewer,

Thank you for your valuable comments, which we have addressed in detail in our response. We would be happy to clarify any further concerns. If our responses resolve your concerns, we would be grateful if you would consider reassessing your score.

Best wishes, Authors

AuthorsOpenReview ↗

Reminder to check our responses

Dear Reviewer 3c4k,

Thank you for your valuable comments, which we have addressed in detail in our response. As the discussion phase is concluding soon, we would greatly appreciate it if you could kindly review our responses and consider reassessing our submission.

Thank you once again for your time and consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Discussion for the AC and Reviewers (1/2)

We thank all the reviewers for their time and constructive feedback. We are encouraged that all the reviewers found the problem important, the use of frozen LLM’s internal signals compelling, and the results promising. The discussion stage helped us to strengthen the experiments and convey the method capabilities more clearly. All these changes have been incorporated into the revised paper, with revisions highlighted in blue.

Discussion status.

  • Reviewer 9EPA responded to our rebuttal, confirming that "the additional experiments address most of my concerns." The remaining points they raised regarding paper formatting have all been addressed in the revised PDF.
  • Reviewer WsvA posted a follow-up comment, which we answered in detail with additional experiments (matched-cost comparison, bootstrap confidence intervals against all baselines, multi-seed significance).
  • Reviewers 3c4k, 6pjY, and JpFi : Their concerns (the inference protocol, the contribution of RL and long-horizon credit assignment, generalization across distractor mechanisms, and end-to-end cost) are addressed in Sections 1–6 below and incorporated into the revised manuscript.
AuthorsOpenReview ↗

Summary of Changes in the Revised Paper (2/2)

Below, we summarize each concern, the reviewers who raised it, how we addressed it, and where the corresponding changes appear in the revised manuscript.

1. Inference protocol and probe questions (Reviewers 3c4k, 6pjY, WsvA, JpFi, 9EPA)

Evaluation questions are not available during memory construction. Sampled probe questions are used only during training and come only from the training split (2:1:7 train/val/test). At test time no probe is provided, so memory is built online from the frozen LLM’s attention over the retained memory and the incoming window. Questions are revealed only after the final window and answered once from the final memory. Section 4.1 now formalizes this with an optional probe (Eq. 1), with the test-time setting stated explicitly. A policy trained with only a generic probe, which never sees a task question, still reaches 0.503 vs. 0.424 for Full Context at 9.2% memory (Appendix E).

2. Contribution of RL beyond attention features (Reviewers 3c4k, WsvA, 9EPA)

We compared CURATE with attention-only selection rules and with a supervised selector, all using the same retrieval heads and features (Section 6.5):

  • At CURATE’s matched budget, attention-only selection reaches 0.185 (threshold), 0.283 (top-k), and 0.139 (random heads), vs. 0.536 for CURATE. The best unmatched threshold reaches 0.419 at 11.7% memory.
  • A supervised CNN on the same features, trained with gold segment labels, reaches 0.489 at 9.7% memory. CURATE outperforms it with less than half the memory.

Beyond the empirical advantage, with the help of RL, CURATE enables learning without knowing the labels of gold vs distractor parts of the context.

3. Contribution of long-horizon credit assignment (Reviewers 3c4k, 6pjY)

A step-reward-only policy matches CURATE at low distractor density but collapses at higher density (12 distractors per question, 0.178 accuracy, under 1% retention). Global reward alone gives 0.482, and combining both gives 0.536 (Appendix B). Both terms are needed. The long-horizon formulation is therefore a necessary design choice, keeping the policy stable under dense distractors where a myopic policy fails.

4. Generalization across distractor mechanisms (Reviewers 6pjY, WsvA, JpFi, 9EPA)

In long-horizon settings, dense distractors are realistic: each retrieval or tool call appends topically close content, so near-misses accumulate over turns. Applied zero-shot, the LoCoMo-trained policy improves over Full Context on two new distractor types: conflicting (+10.4 points, 9.1% memory) and retrieval-based from real dialogue (+5.7 points, 5.3% memory). AttentionRAG gains at most 1.6 points while keeping about 47% of the context (Section 6.6).

On unmodified LoCoMo, CURATE matches the strongest baselines (0.554) at roughly half the memory, and it degrades the least when distractors are added (Appendix A).

We also clarified the LongMemEval results and why retention there (50.1%) is higher than on LoCoMo (Appendix C.1).

5. Matched budgets and memory accounting (Reviewers WsvA, 3c4k)

We ran every baseline whose budget can be controlled at CURATE’s budget and revised our memory accounting. Our observations are as follows:

  • At every memory budget, CURATE is more accurate than every baseline. At 4.3% memory, it surpasses methods that keep the entire context, such as Full Context (0.424) and MEM1 (0.427), and at a similar budget (≤7%) it is 19.0 points above the strongest baseline, MemAgent (Appendix D).
  • We corrected the MemAgent results in Table 2 (0.346 accuracy, 6.1% memory).

6. End-to-end cost (Reviewers WsvA, JpFi, 9EPA)

The reported latency includes the full cost of incremental memory maintenance. End-to-end latency is about 1.1 s per query. Appendix D reports LLM calls, tokens, and latency for all methods (Table 11) and a per-stage breakdown for CURATE (Table 12). Compared with memory systems that use an LLM to write or consolidate memory, CURATE processes 3 to 60 times fewer LLM tokens per query and is 4.5 to 51 times faster.

7. Additional backbone, statistics, and related work (Reviewers WsvA, 9EPA)

We conducted additional experiments with a second backbone (Llama-3.1-8B), added uncertainty estimates, and compared with Memory-R1 trained under the same distractor setting.

  • With Llama-3.1-8B, CURATE reaches 0.524 vs. 0.466 for Full Context at 6.9% memory (Appendix C.3).
  • Paired bootstrap confidence intervals exclude zero for the gain over 11 of the 12 baselines, and CURATE remains ahead of MEM1 in 95.3% of resamples. Over six seeds, CURATE reaches 0.523 ± 0.013 (Appendix H).
  • CURATE is 10 points above Memory-R1 (0.536 vs. 0.436) with less than 5% of the context (Table 2).

We believe these additions address the concerns raised and strengthen the paper. We again thank all reviewers for their valuable feedback and for helping us improve this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.