- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a context selection framework CURATE based on reinforcement learning for long-term sequence LLM agents. This method first identifies a set of fixed retrieval heads from the frozen LLM and uses their attention patterns to represent the relevance of the input context segments. Subsequently, a lightweight MLP or CNN strategy makes decisions on whether to retain or discard each segment. This strategy is trained using relative policy gradients within the group and uses answer correctness and memory cost as evaluation metrics, without the need for explicit relevance labels.
Critical strengths
- An interesting method that utilizes internal model signals for context selection.
- It has strong empirical results and practical analysis.
Critical weaknesses
- It is not clear whether there are probe questions during the online scenario selection process.
- The contribution of the optimization method based on reinforcement learning compared to the heuristic method based on attention has not been fully separated.
Questions for authors
Q1. Probe Question Availability.
During the training and evaluation processes, it is important to understand how the probing questions are sampled, especially for LoCoMo. Does the selection strategy assess the standard benchmark questions that are only raised in the later stages of the conversation? Additionally, how does this method identify which questions in the current memory have sufficient evidence without relying on gold evidence annotations?
Q2. Contribution of RL-based Context Selection.
Could the author use the same retrieval head, question formulation, and memory budget to compare CURATE with a simple attention-based ranking baseline? Since Table 1 already shows a significant separation between the gold segments and the interfering segments, such a comparison would help demonstrate the additional advantages brought by learning the selection strategy through reinforcement learning.
Q3. End-to-end Computational Efficiency.
The reported delay of 1.6 seconds per query in Table 5 includes whether it encompasses the repeated detection generation and attention extraction forward propagation required during the memory construction process? Can the authors report the complete scene-level or end-to-end costs, separately explaining the expenses for feature extraction, policy inference, and final answer generation?
Q4. Baseline Comparison and Generalization.
Memory-R1 [1] is a memory management method based on reinforcement learning, which has been evaluated on LoCoMo and LongMemEval. Can the authors discuss its relationship with CURATE and provide a comparative analysis under the condition-matching setting (if feasible)?
In addition, CURATE retains 50.1% of the context on LongMemEval, while LoCoMo only retains 4.3%. Can the authors further analyze this difference and evaluate this method in the context of natural occurrence interference items beyond the synthetic generation process?
Reference
[1] Yan et al., "Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning." ACL 2026. https://aclanthology.org/2026.acl-long.583/
9 comments
Part 2: Response (2/3)
Q3 (End-to-end Computational Efficiency) :- Yes, the latency in Table 5 includes the repeated trace generation and attention-extraction forward passes during memory construction. At every selection step, the frozen LLM generates a 16-token trace and runs one forward pass to read attention, and both are counted together with policy inference and final answering. We report the per-query and per-episode costs on LoCoMo, separately for feature extraction, policy inference, and final answer generation:
| CURATE, end to end | Avg per query | Avg per episode |
|---|---|---|
| LLM calls for selection (trace generation + attention pass) | 8.6 | 33.5 |
| LLM calls for Answer generation | 1 | 3.9 |
| Total processed prompt tokens (selection + answering) | 3,013 | 11,749 |
| Latency: feature extraction | 1.02 s | 3.97 s |
| Latency: policy inference | 0.01 s | 0.05 s |
| Latency: final answering | 0.08 s | 0.32 s |
| End-to-end latency | 1.11 s | 4.33 s |
We re-measured the complete pipeline on a dedicated GPU with no other jobs running. Over three executions the end-to-end latency is 1.126 ± 0.01 s per query (1.11, 1.14, 1.13 s), consistent with and slightly below the 1.6 s in Table 5, where the timing is sensitive to GPU load. The breakdown is stable across runs: feature extraction accounts for about 92% of the time, final answering for 7%, and policy inference for 1%. The end-to-end latency remains an order of magnitude below the memory systems that call an LLM to write or consolidate memory at every turn. We will add this breakdown in the paper.
Q4 (Memory-R1 Comparison and Generalization):-
Memory-R1 manages memory with two RL-fine-tuned LLM agents: a Memory Manager that extracts facts from the dialogue and issues ADD / UPDATE / DELETE / NOOP operations on a memory bank, and an Answer Agent that filters the retrieved memories before answering. Both methods use outcome-driven RL, but they differ in what is trained and what is stored. Memory-R1 fine-tunes the full LLM (7-8B parameters) and stores rewritten facts; CURATE keeps the LLM frozen, trains only a 378K-parameter policy on the model's own attention, and makes the decision to keep or discard the segments.
We trained Memory-R1 using the authors’ code under the same distractor setting on LoCoMo (GRPO, Qwen2.5-7B).
| Method | Trained parameters | Acc | Context given to the LLM |
|---|---|---|---|
| Full Context | – | 0.424 | 100% |
| Memory-R1 | 7B (full LLM) | 0.436 | 91% |
| CURATE (CNN) | 378K | 0.536 | 4.3% |
Memory-R1 performs on par with Full Context, while CURATE is 10 points higher with less than 5% of the context. Because the Memory Manager extracts facts from every turn, distractor turns are also stored and later retrieved for the reader. Memory-R1 learns how to organize memories, but it has no mechanism to recognize a stored statement as a distractor. The two approaches are therefore complementary: CURATE’s selection can be applied before a memory manager writes to its bank.
Retention on LongMemEval (50.1%) vs. LoCoMo (4.3%). The difference comes from the composition of the two benchmarks. On LoCoMo, distractors make up 74% of the segments, so most of the context can be removed; on LongMemEval they are only 12% of the segments (under 2% of the tokens), and the remaining 85% is genuine conversation, so even a perfect distractor filter retains 88% (Oracle). CURATE still retains much less than that (50.1%): beyond the distractors, it also learns to discard redundant content along with it.
Part 3: Response (3/3)
Q4 (Generalization across Distractor mechanisms) :- In practical RAG and agentic memory pipelines, distractors arise naturally: retrievers surface passages that are semantically similar to the query but misleading. Recent work shows that such distractors capture attention even at small proportions, so that even a small fraction of them sharply degrades long-context performance, and recovery requires removing them almost entirely (Gao et al., The First Drop of Ink, ICML 2026). Our retrieval-based distractors (2) reproduce this setting directly. To show generalization across different distractor mechanisms, we compared the trained CURATE policy with Full Context and AttentionRAG on two independently constructed distractor types, matching the distractor density. Both are built on the LoCoMo test conversations using the same pipeline as in the paper, with only the distractors replaced. In both cases, the LoCoMo-trained CNN policy is applied zero-shot.
(1) Conflicting distractors. The gold speaker restates the same fact with a conflicting value, so the context contains an explicit contradiction of the evidence. For example, the gold turn by John states, “we go a few times a week,” while the inserted distractor, also attributed to John, states, “I take my kids to the park every day.” Unlike the distractors in the paper, which attribute the competing detail to a third party, these directly contradict the gold evidence.
(2) Retrieval-based distractors. Instead of generating distractors, we retrieve them: for each question, a dense retriever returns the most similar real dialogue turns from other conversations, which are then inserted into the session. For example, for the gold turn by John, “We had to say goodbye to Max. He was such an important part of our family for 10 years,” the retriever returns a real turn by Maria from another conversation, “Max is my mother's cat, I took him when my mother passed away.” These mirror the hard negatives a RAG pipeline would surface; they are topically similar, naturally written, and not produced by any generator.
| Distractor type | Method | Acc | Mem | G/D |
|---|---|---|---|---|
| Contradictory | Full Context | 0.369 | 1.000 | 1.00 |
| Contradictory | Random (matched budget) | 0.096 | 0.093 | 1.08 |
| Contradictory | AttentionRAG | 0.385 | 0.478 | 1.31 |
| Contradictory | CURATE (CNN) | 0.473 | 0.091 | 52.23 |
| Contradictory | Oracle (no distractors) | 0.521 | 0.526 | – |
| Retrieval-based | Full Context | 0.466 | 1.000 | 1.00 |
| Retrieval-based | Random (matched budget) | 0.063 | 0.054 | 0.83 |
| Retrieval-based | AttentionRAG | 0.476 | 0.474 | 1.23 |
| Retrieval-based | CURATE (CNN) | 0.523 | 0.053 | 72.80 |
| Retrieval-based | Oracle (no distractors) | 0.560 | 0.270 | – |
Both mechanisms degrade Full Context (15.2 and 9.4 points below the Oracle). Applied zero-shot, CURATE improves over Full Context by 10.4 and 5.7 points while retaining only 9.1% and 5.3% of the context. AttentionRAG gains at most 1.6 points, keeps about 47% of the context, and retains distractors almost as often as gold evidence (G/D ≈ 1.3). CURATE therefore generalizes to distractor mechanisms it was not trained on, without requiring retraining. We will add these results to the revised paper.
Reference:- Gao et al. The First Drop of Ink: Nonlinear Impact of Misleading Information in Long-Context Reasoning. ICML 2026. arXiv:2605.10828.
Part 1: Response (1/3)
We sincerely thank the reviewer for their thoughtful feedback and constructive evaluation of our work.
W1, Q1 (Probe Question Availability) :- Evaluation questions are never available during memory construction. The sampled probe questions are used exclusively during training, as indicated in the RL Training Stage of Figure 1, and are drawn only from the training split. We use a split of 2:1:7 ratio (train, val and test), so no test conversation or question is seen during training. At test time, no probe question is provided: the attention features are obtained directly from the frozen LLM’s forward pass over the retained memory Mt−1 and the incoming window Ct. All test questions are answered only once, against the final memory MT.
During training, a question enters the probe pool once the turns containing its evidence have arrived. The benchmark’s evidence positions are used only for this scheduling. They are never given to the policy, which does not observe which segments are evidence, and the reward depends solely on whether the frozen LLM’s answer matches the reference. To remove this dependence entirely, we also trained CURATE with a fixed generic probe (Appendix G.6), so the policy never sees a task question when computing its state. It still reaches 50.3 against 42.4 for Full Context while retaining 9.2% of the context (83.9% of gold vs. 2.3% of distractors kept). Training with question-specific probes therefore improves accuracy (53.6 vs. 50.3), but the policy learns to retain gold evidence and discard distractors even without access to any task question.
W2 , Q2 (Contribution of RL-based selection):- As suggested, we compared CURATE with attention-only selection using the same retrieval heads, question formulation, and windowing, with no learned policy. Top-k ranking keeps the highest-scoring segments by retrieval-head attention, and threshold keeps segments whose attention exceeds a fixed value; both are set to CURATE’s memory budget. We also report the best threshold without the budget constraint.
Attention-only selection is itself a strong signal: with its best threshold, it reaches 0.419 while retaining only 11.7% of the context, close to Full Context at a fraction of the memory. This confirms that the retrieval heads carry useful relevance information. At a matched budget, however, CURATE clearly outperforms it (0.536 vs. 0.283), with a much higher G/D (100.2 vs. 11.58). The RL policy therefore makes better use of the same attention features, turning them into more reliable keep/discard decisions under a tight memory budget.