acceptodds
Under review as a conference paper at ICLR 2027

Credit Follows Use: Used-Grouped Policy Optimization for Long-Context Memory Agents

Abstract

Long-context memory agents have emerged as a promising paradigm for retaining information across document streams and multi-session interactions. However, optimizing these agents poses a fundamental credit-assignment challenge: the consequences of a memory operation may remain unobserved until its content is retrieved and grounded in a distant response. Existing approaches typically either propagate a trajectory-level reward to all operations or derive process-level supervision from gold evidence and branched rollouts. We argue that both paradigms overlook an observable signal intrinsic to external memory systems—each materialized memory item exhibits a traceable lifecycle from writing, through retrieval, to grounded use in downstream generation. Inspired by this observation, we propose Use-Grouped Policy Optimization (UGPO), which records Write–Retrieve–Use traces and organizes policy updates around recurring item-use events rather than complete memory states. By modeling this lifecycle, UGPO routes task, use, and cost credit to responsible operations and propagates delayed outcomes to their originating writes. We further characterize state-group collapse, where free-form memory updates progressively eliminate state-level comparison groups, while item-use groups remain comparable across diverged memory states. Extensive experiments on long-document and multi-session benchmarks demonstrate that UGPO consistently improves task performance and attribution quality while reducing memory pollution and serving cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Memory Agent, Reinforcement Learning, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KK7gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose UGPO (Use-Grouped Policy Optimization), a framework for credit assignment in long-context memory agents. Instead of broadcasting a trajectory-level reward to every memory operation, the method records the observable lifecycle of each memory item, from writing through retrieval to grounded use in a later response, and routes task, use and cost credit only along these paths. The comparison is done at item-use level, not under exact memory-state equality, because the authors show with a small bound that grouping by exact state lose coverage exponentially when more free-form writes are accumulated. Rewards are normalized for each channel before aggregation, and the delayed uses are replayed to supervise the writes they come from, so there is no need to regenerate the interaction history. Experiments are done on four benchmarks with nine baselines under same budget.

Critical strengths

For me, the theoretical part is a real contribution. The coverage bound for exact-state grouping is simple and clear. It shows where the state-relative methods can be applied and where not in the free-form memory setting. The new grouping key is designed to not include any sampled memory content or reward. This removes the circularity concern, that the groups are built from the outcome which they should predict.

The evaluation protocol is also very complete. The authors fix the three reward channels and only change the assignment operator, so the source of gain can be separated. For statistics, all tables report five seeds with standard deviation, and the main comparison use paired bootstrap intervals with Holm correction. The authors also build failure cases for their own method. In addition, the rollout multiplier keep at one since all supervision comes from the original trace. This is a structural advantage on efficiency compared to the baselines that need branched continuations.

Critical weaknesses

All differences are statistically significant, but compared to the strongest baseline, the improvement on macro average is less than one point, and on each single benchmark it is at most 1.2. The large improvements are mainly on attribution and pollution metrics. So the practical value of this framework depend on how much a real system can benefit from more precise credit and cleaner store, not on the task numbers. For readers who mostly care about task improvement, I am not sure this trade-off is explained clear enough.

Questions for authors
  1. For a practitioner who already have a memory agent in service, what is the benefit on task performance if they switch to UGPO? Are the improvements on attribution and pollution worth the extra cost of trace infrastructure and the tuning of multiple channels in real deployment?

  2. The grouping key is a combination of item buckets, request strata and delay bins, and there are many design choices in how to combine them. How did you decide this combination? When applying to a new workload, how much of the key need to be redesigned? Besides the sensitivity scan, is there a more principled way to choose the key?

Reviewer_bjz2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies credit assignment for long-term memory agents, where a write may become useful only much later and a shared trajectory reward cannot distinguish contributing operations. UGPO records stable memory identifiers, update lineage, writes, retrievals, citations, and task outcomes. An item counts as grounded use only if it is retrieved, cited, and accepted by a frozen NLI support verifier. Successful grounded use routes task credit to the associated write and retrieval. If supporting memory is followed by a failed answer, the negative task feedback stays with answer generation rather than the earlier write.

The method compares items retrieved for the same request and forms replay groups of historical events using source clusters, request strata, and write-to-use delays. Task, use, and storage/retrieval cost rewards are normalized separately within groups, combined into an advantage, and applied to the responsible operation tokens. Stored behavior-policy probabilities allow later-session outcomes to update earlier writes without rerunning intervening interactions. The main experiments use Qwen2.5-7B on HotpotQA, 2WikiMultiHopQA, LoCoMo, and LongMemEval, measuring task performance, attribution, and system cost, with additional 14B/32B and ALFWorld/WebShop results. The reported four-task average exceeds HiMPO by 0.9 points. Table 14, which fixes rewards and changes assignment, reports a 1.3-point gain over write-local contrast.

Critical strengths
  1. Requiring retrieved items to be cited and pass a support check can reduce credit assigned to irrelevant context. Identifiers and update lineage connect this feedback to specific writes. Delayed replay lets use observed much later contribute to training without rerunning the intervening history. These choices address the concrete difficulty of delayed feedback in persistent memory.

  2. Table 14 uses identical task, use, and cost rewards across assignment operators, helping separate richer rewards from the assignment mechanism. Table 3 removes grounded use, grouping, replay, and normalization individually; Table 12 perturbs grouping keys; and Table 13 reports paired comparison intervals. Means and standard deviations over five seeds provide further evidence for assessing the components.

Critical weaknesses
  1. Table 5 on p.17 specifies an 8K retained-token budget for every method, and p.18 states that the appendix uses the same budgets. Table 6 nevertheless reports store sizes of 9,800 and 9,100 tokens for Outcome GRPO and GiGPO, explicitly defined as all retained tokens at evaluation time. Under these definitions, the rows conflict with the capacity limit. If these are pre-truncation measurements, different accounting units, or different capacities, that needs to be stated, followed by checking task performance, pollution, and latency at matched capacity. UGPO pollution also rises during training in Figure 6(b) on p.19, whereas the immediately following text says it falls. The curve or its description needs clarification.

  2. Appendix B infers a probability of matching after n writes of at most gamma^n from the assumption that currently matching stores remain matched after the next write with conditional probability at most gamma. This does not follow: the product bounds remaining matched at every intermediate step, not matching again after earlier divergence. The operation space includes updates and deletions, but no assumption rules out reconvergence. Eq. (15), and consequently Eq. (4), are therefore not established as written. The argument needs additional conditions or a revised bound.

  3. Section 4.1 and Appendix H do not specify the actual training, validation, and test splits or sample counts for the four primary datasets, and the document-stream construction is insufficiently detailed. Appendix H maps gold evidence to the unique memory ID of the write that first materialized it using source documents and character offsets, without explaining paraphrases, consolidation, or repeated writes. Appendix E does describe credit propagation through update and deletion lineage; what remains unclear is how that lineage corresponds to evaluation reference IDs. Since Table 10 uses this set directly for attribution precision and recall, the fairness of the evaluation across writing policies cannot yet be independently checked.

Questions for authors
  1. Is Table 6 store size measured before or after applying the 8K limit, with which tokenizer and aggregation? How does this correspond to Table 5, and which task or efficiency results need revision at matched capacity? How should the disagreement between Figure 6(b) and its discussion be corrected?

  2. Does Eq. (15) additionally assume that stores cannot reconverge, or directly constrain the collision probability after each write? How does that apply to updates and deletions? Please provide the corresponding proof or a corrected Eq. (4).

  3. Please specify dataset splits, sample counts, and stream construction. Could you show a complete attribution-evaluation example involving a paraphrased or updated memory, including source evidence, identifier lineage, the reference set, and credited operations?

1 comment

Dear Reviewer bjz2,

We sincerely thank you for your rigorous review and valuable suggestions.

If our rebuttal has addressed your concerns, we kindly hope you might reconsider your evaluation of our paper.

Wish you a wonderful life.

Best regards,

Reviewer_1fjeOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies credit assignment for long-context memory agents, where a memory write may only become useful much later, after retrieval and grounding in an answer. It proposes Use-Grouped Policy Optimization (UGPO), which records Write-Retrieve-Use traces, detects grounded memory use with a frozen entailment verifier, routes task/use/cost rewards to the responsible operations, forms listwise and replay item-use groups rather than exact memory-state groups, and supports delayed-use replay to train the originating writes. The paper argues that exact state grouping rapidly loses comparison coverage as free-form memory diverges. Across HotpotQA, 2WikiMultiHopQA, LoCoMo, and LongMemEval, UGPO improves the reported macro-average over the strongest listed baseline (HiMPO) from 67.7 to 68.6, while also improving attribution precision, write utilization, retrieval precision, and memory pollution under matched settings. Ablations and scaling experiments support contributions from item-use grouping, responsibility routing, channel-wise normalization, and replay.

Critical strengths
  1. A practical lifecycle-based credit-assignment formulation. The explicit Write--Retrieve--Use trace is a natural way to connect delayed downstream outcomes to memory operations without requiring additional rollout branches or gold evidence labels during training. This addresses a real bottleneck in learning memory policies.

  2. Broad, matched evaluation with useful diagnostics. The paper evaluates four memory benchmarks, multiple model sizes, task quality, attribution, pollution, retrieval efficiency, latency, and delayed-use behavior. The same-reward/different-assignment ablation is particularly valuable because it isolates item-use grouping from simply adding more reward channels.

Critical weaknesses
  1. Verifier- and provenance-dependence weakens the attribution interpretation. Grounded use is defined by retrieval, citation, and a frozen NLI verifier, while reference provenance is constructed from dataset annotations and source offsets. Errors in entailment, citation localization, stale-memory handling, or source alignment can change both the training signal and the reported attribution precision. The reported gains therefore do not yet establish that UGPO assigns causal responsibility rather than optimizing a particular proxy.

  2. The generality of the grouping argument is narrow. UGPO improves the macro-average by 0.9 points over HiMPO in the main comparison, while introducing several coupled mechanisms and fixed coefficients. The coverage theorem applies to exact, high-specificity full-store keys under a particular divergence assumption; it does not rule out competitive coarser state abstractions or establish that item-use grouping is preferable across other memory architectures. More direct controls are needed to separate the contribution of the proposed grouping from the verifier, reward design, and replay.

  3. The author needs to adopt a more standardized approach when citing references. When the cited object is a sentence component (such as the subject or object), \citet should be used; when it is not a sentence component, \citep should be used.

Questions for authors
  1. How sensitive are task performance, attribution precision, and pollution to the frozen NLI verifier, its threshold, and the choice of verifier model? In particular, do the main conclusions hold with at least one independent verifier or with human-checked subsets?

  2. Can you provide an ablation that keeps the same WRU traces and all three reward channels but replaces the grounded-use predicate with a less verifier-dependent signal, and an ablation that removes delayed-use replay while retaining item-use grouping? These results would clarify which mechanism accounts for the improvement over HiMPO.

  3. How are conflicting, stale, or multiply-supported memory items handled in the benchmark data, and how often do the cited-span and provenance rules disagree? Please report these rates and their effect on the main results.

  4. The main comparisons use fixed channel weights and a fixed grouping construction. Were these choices selected using a development split, and how robust are the conclusions to reasonable changes in the channel weights, group-key hyperparameters, and retrieval depth?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.