acceptodds
Under review as a conference paper at ICLR 2027

Identify Then Reinforce: Action-Level Credit for Graph-Structured Memory Agents

Abstract

Personal memories are interdependent: new experiences can update earlier facts, introduce conflicting claims, or connect to related information from distant interactions. Existing learnable memory systems primarily operate on flat textual entries, leaving these relationships implicit and making it difficult to reconcile evolving information and retrieve related evidence across time. We introduce PersonaMap, a relational memory framework that learns how personal memories should be extracted, organized, and accessed. To learn memory construction and retrieval jointly, we propose a shared policy strategy that extracts atomic claims from interaction histories, constructs semantic relations, and navigates the Personal Memory Graph through tool interactions. To distinguish the contributions of individual memory actions, we infer action-level credit from outcome differences across natural rollouts and invoke counterfactual rollouts only when natural evidence is insufficient to identify an action's effect. PersonaMap further supports structure-aware forgetting that prunes weakly connected claims while preserving strongly connected memories within the graph. Across two personalized long-term memory benchmarks, PersonaMap improves overall accuracy over the strongest baselines by 7.68% and 4.29% respectively. Controlled comparisons support the benefit of action-level credit. With 50% fewer stored claims, structure-aware forgetting retains complete mapped relational evidence for approximately 80% of eligible questions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM agents, long-term personalized memory, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_HrTxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PersonaMap, a personal memory agent that stores atomic claims in a graph with four typed relations (same, complements, unresolved conflict, distinct-by-qualifier) and navigates it with tool calls. One LoRA policy on Qwen2.5-7B-Instruct does all stages and is initialized by SFT on GPT-5.1 teacher traces that were converted into the tool protocol. RL is done with RICO, a GRPO variant that gives credit only to actions at "natural forks" (matched prefix and state, different actions, different mean terminal reward). When a rollout group has no such fork it falls back to a small counterfactual replay. The reward is a binary evidence-sufficiency verdict from a frozen Qwen3.8-27B judge on the claims the agent observed. A deterministic structure-aware pruning rule is also proposed. Results: 50.81% on SubtleMemory (personas 7 to 9), 56.41% zero-shot on PersonaMem, RICO 50.81 vs GRPO 48.12 from the same SFT checkpoint, and about 80% relational evidence retention at 50% claim compression.

Critical strengths

S1: Credit assignment matched to the structure of memory actions. Restricting credit to decisions where matched-prefix branches actually disagree, and replaying a counterfactual only when needed, is a reasonable and cheap way to avoid smearing one outcome over extraction, relation and navigation steps. The coverage analysis (Fig. 4, 33.2% of questions get usable credit, 23.7% from natural forks) is useful because it shows how sparse the signal really is.

S2: Policy/executor split. The policy picks content and labels while the executor binds endpoints and mutates the graph, and teacher traces are replay-validated (A.6). This makes the system easier to reason about than most LLM-memory pipelines.

Critical weaknesses

W1: The RICO vs GRPO evidence is thin. The method claim rests on 50.81 vs 48.12 in Table 2. That is a single run, with no seeds or intervals, on three held-out personas whose question count I could not find. The compute is also not matched. "Same rollout budget" in Fig. 5 seems to count only the K=16 base rollouts, while RICO adds counterfactual replays and suffix samples. With M=2 suffixes per branch, the counterfactual credit takes values like plus or minus 0.25 or 0.5 and is very noisy. I would want several seeds, GRPO with matched total samples, and a variant that turns off the counterfactual fallback (natural forks only). Fig. 5 (smaller advantages and gradient norms) describes the update. It does not tell me the update is better.

W2: Main comparison is confounded. PersonaMap is distilled from GPT-5.1 teacher traces built from SubtleMemory personas 0 to 6, i.e. the same distribution as the test personas. The paper does not say which LLM backs the training-free baselines or what data the trainable ones were trained on (I may have missed this, but Table 1 gives no backbone column). Some numbers look off: Full Context at 4.27% and most baselines near 0 to 4% on contradictory questions. That makes me suspect a protocol mismatch more than a real gap. The evaluation answer judge for SubtleMemory is also not described. The training reward never scores the generated answer at all (only evidence sufficiency), so I would like to see how well reward and test accuracy line up. The ablation also shows SFT alone (45.88) below the fixed structured pipeline (46.99), so most of the gain over baselines comes from the teacher-derived graph design rather than from RL.

One more: the forgetting result is evaluated on personas 0 to 6 (training personas), only as a graph-level proxy, with no end-to-end accuracy (Appendix C says this is pending). The proxy requires the gold relations to survive, and the method protects anything reachable through relations, so it is partly favored by construction. The abstract claim should be softened.

Questions for authors

Q1. How many test questions are in SubtleMemory personas 7 to 9? Can you report mean and std of RICO and GRPO over at least three seeds, and GRPO at matched total rollout count including the counterfactual samples?

Q2. Which backbone LLM and which checkpoints were used for each baseline in Table 1? How is answer accuracy judged at evaluation time? What is the Structured Graph row exactly (GPT-5.1 or the 7B model running the fixed pipeline)?

Q3. In practice, at what depth do natural forks occur? I would guess that exact prefix plus state fingerprint matching at K=16 mostly gives forks at the first divergent action. If so, how much of the credit actually lands on relation and navigation decisions rather than early AddClaim calls?

Q4. How often does the evidence-sufficiency reward disagree with final answer correctness on held-out data?

2 comments
AuthorsOpenReview ↗

Response to reviewer HrTx

Thank you for the constructive feedback. We appreciate your recognition: our credit-assignment design is a reasonable and inexpensive way to avoid conflating outcomes across memory operations; the coverage analysis is useful; and the policy/executor split makes the system easier to reason about than most LLM-memory pipelines. We address your concerns below.

1) Question counts, RL comparisons, and sampling budgets (W1, Q1).

We apologize for the aggregation and reporting error. Earlier SubtleMemory results used incomplete aggregates that did not consistently cover all three held-out personas. We have completed evaluation on all 422 questions from personas 7–9: 107 complementary, 207 nuanced, and 108 contradictory. Missing or invalid predictions count as incorrect.

MethodCorrect/totalAccuracy
Tool-Joint SFT205/42248.58%
SFT + GRPO209/42249.53%
SFT + RICO224/42253.08%

These results replace the earlier aggregates. RICO improves over the shared SFT initialization by 4.50 percentage points and over GRPO by 3.55 points in this run.

“Same rollout budget” refers to the K=16K=16 base rollouts. RICO first reuses natural comparisons within these rollouts, then selectively adds CF continuations only for groups without usable natural credit. This increases usable-credit coverage from 23.7% to 33.2% of questions. This additional signal requires additional sampling: with BCF=1B_{\mathrm{CF}}=1 and M=2M=2 continuations per branch, an eligible group can require up to four extra suffix continuations. Thus, equal base-rollout budgets do not imply equal total sampling cost.

The reported QA results are single-run comparisons. Additional seeds and total-sampling-matched GRPO controls are currently underway. These comparisons require multiple independent RL training runs and substantial GPU resources, making them time-consuming. We will supplement the results and analysis once the runs are complete and verified.

We agree that smaller advantages and gradient norms alone do not establish better updates. Fig. 5 illustrates RICO’s update characteristics; its downstream benefit is evaluated through the QA comparison.

2) Baseline models, evaluation, and the structured pipeline (W2, Q2).

All Table 1 scores come from our own evaluation runs. Appendix D.1 describes the evaluation protocol, and Appendix D.2 specifies the baseline configurations. Training-free memory baselines—Mem0, Mem0-g, A-Mem, LightMem, MemoryOS, and GAM—use Qwen2.5-7B-Instruct for memory operations, with the 128K variant on SubtleMemory. MemAgent, Mem-α, UMA, and MEM1 use the RL-MemoryAgent-7B, Memalpha-4B, UMA-4B, and MEM1-Qwen2.5-7B checkpoints, respectively. Memory-R2 uses its own memory-management checkpoint. The shared answer-generation pipeline uses Qwen2.5-7B-Instruct at temperature zero (Appendices D.1–D.2).

The Structured Graph baseline uses Qwen2.5-7B-Instruct to run our fixed pipeline. For SFT data construction, GPT-5.1 runs the pipeline to generate teacher demonstrations (Appendix A). Thus, the Structured Graph score reports the Qwen-based baseline, rather than the GPT-5.1 teacher’s performance.

For SubtleMemory, answer correctness follows the benchmark’s evaluation procedure, using Gemini-2.5-Flash with majority voting over three calls when LLM judgment is required. The answer prompts and scoring procedures are detailed in Appendix D.3.

Responses to the remaining concerns continue in Part 2/2.

AuthorsOpenReview ↗

Response to reviewer HrTx

This continues our response to Reviewer HrTx from Part 1/2.

3) Natural-fork depth and credited action types (Q3).

Exact prefix matching means a matched branch point is the first divergence of the compared trajectories. It can occur at different stages, depending on how long their prefixes agree.

Fig. 4 includes credited nodes for ClassifyRelation, Neighbors, and Inspect, showing that credit reaches relation and navigation decisions as well as extraction. These are credited-node counts, not unique-question counts or a depth distribution. We do not currently have a depth-by-stage breakdown. Repeated-action averaging and matched-context CF comparisons improve comparability, but the small continuation budget leaves finite-sample uncertainty.

4) Evidence reward versus answer correctness (W2, Q4).

The frozen Qwen3.8-27B judge evaluates whether the observed evidence supports answering the question; it does not score the generated answer (Appendix B.3). Final QA scoring is performed separately. Sufficient evidence can still lead to an incorrect answer, so these signals need not agree. We have not measured their held-out disagreement rate and cannot report a verified percentage.

5) Forgetting scope and downstream QA (W2).

The original approximately 80% result measures complete mapped relational-evidence retention on eligible questions from personas 0–6 at 50% claim compression. It is a graph-level result, not held-out QA accuracy. The pruning operator uses graph structure and does not access gold evidence mappings, although the structural metric alone cannot establish downstream utility.

We additionally evaluated QA after removing 50% of claims on the same 422 held-out questions. Accuracy decreases from 53.08% (224/422) to 39.10% (165/422), retaining 73.7% of its unpruned value. This directly quantifies the downstream cost of compression.

Forgetting is currently an offline, deterministic operation; longitudinal memory-growth and maintenance-cost curves have not been evaluated. We will revise the abstract to state the scope of the evidence-retention result explicitly.

Reviewer_yG5UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PersonaMap, a memory system that represents information about a user as a graph of short claims. Relations connect claims that are equivalent, complementary, conflicting, or different because of time or context. A shared Qwen2.5-7B-Instruct policy extracts claims, predicts their relations, and navigates the graph to answer questions.

The policy is first trained on demonstrations produced using GPT-5.1, then refined with RICO. RICO estimates action-level credit by comparing actions taken in matching contexts. When a rollout group provides no usable natural comparison, it runs additional counterfactual continuations. The system also includes a fixed pruning rule. The authors report 50.81% accuracy on SubtleMemory, 56.41% on PersonaMem, and a 2.69 percentage-point improvement over GRPO

Critical strengths
  1. The paper addresses an important problem: personal information changes over time and may contain unresolved conflicts. Keeping conflicting claims accessible is a sensible design.

  2. The separation between learned semantic decisions and deterministic graph operations makes the system easier to understand.

  3. Reusing existing rollout comparisons before generating additional continuations is a useful way to limit extra sampling.

Critical weaknesses
  1. Test set sizes are not reported, and methods may not share the same questions. The paper never states how many test questions are used, overall or per category. The reported numbers also suggest a possible mismatch: all baseline values on SubtleMemory match category sizes of 107, 207 and 108 questions (e.g., Full Context's 4.67% =5/107= 5/107), but PersonaMap's do not (no whole number out of 107 gives 51.56%). Similar mismatches appear on PersonaMem and in Table 2. This is only an inference from rounded numbers, so I ask the authors to report question counts per method and category, and whether any questions were excluded.

  2. It is unclear why RICO beats GRPO. Only 33.2% of questions produce any signal and about 2% of tokens are updated. The gain may simply come from changing the model less after SFT. A GRPO baseline with matched update size, and results over several runs, are needed; the 2.69-point gap comes from a single run.

  3. RICO's credit may go to the wrong action, and this is never checked. Rollouts are compared only at their first divergence, so any later difference that actually caused the reward change is still credited to the first one. With two or three rollouts per choice, the estimate is very noisy, and later steps rely on a small replay budget (BCF=1B_{CF} = 1, M=2M = 2). No experiment verifies that credited actions are actually better, and it is unclear how alternatives are chosen for free-text actions such as AddClaim.

  4. Relation label quality is not measured, and all results rely on one set of models. Labels such as Conflicts vs DistinctByQualifier are often ambiguous, and the GPT-5.1 training labels are checked only for executability, not correctness. All experiments use the same teacher, base model and judge, so it is unclear how much of the gain comes from distilling GPT-5.1 and whether results hold with other models.

  5. Memory only grows. The model adds, links and reads claims but never updates or deletes them; deletion is a fixed rule whose effect on accuracy is not tested. This should be stated clearly, along with how graph size, accuracy and cost change as conversations accumulate.

Questions for authors
  1. How many test questions are there per method and category? Were any excluded?
  2. How does RICO compare with a GRPO baseline of similar update size, over several runs?
  3. Does RICO's credit stay the same when re-estimated with many more runs?
  4. How accurate are the relation labels from GPT-5.1 and from the trained model?
  5. How are the starting claims for navigation chosen, and how often are they enough to answer on their own?
4 comments
AuthorsOpenReview ↗

Response to reviewer yG5U

Thank you for the constructive feedback. We appreciate your recognition: our work addresses an important problem; keeping conflicting claims accessible is a sensible design; the policy/executor split makes the system easier to understand; and reusing rollout comparisons is a useful way to limit extra sampling. We address your concerns below.

1) Question counts and evaluation comparability (W1, Q1).

We apologize for the aggregation and reporting error, and thank you for your careful checking. The earlier reporting used incomplete aggregates that did not consistently cover all three held-out personas.

We have now completed scoring on the full SubtleMemory test set from personas 7–9, comprising 422 questions: 107 complementary, 207 nuanced, and 108 contradictory. All three variants are evaluated on these same questions, with missing or invalid predictions counted as incorrect.

MethodComplementaryNuancedContradictoryOverall
Tool-Joint SFT53/107 (49.53%)128/207 (61.84%)24/108 (22.22%)205/422 (48.58%)
SFT + GRPO54/107 (50.47%)136/207 (65.70%)19/108 (17.59%)209/422 (49.53%)
SFT + RICO62/107 (57.94%)140/207 (67.63%)22/108 (20.37%)224/422 (53.08%)

Each entry reports correct/total questions and accuracy. These results replace the earlier incomplete aggregates. All SubtleMemory baselines also use the complete 422-question set.

For PersonaMem 32K, all methods load the same 589 questions from 20 personas. The original aggregation calculates accuracy over predictions with parseable answer options, excluding processing failures and empty or invalid outputs. We therefore report each method’s scored count and exclusions, alongside accuracy over all 589 questions with unscored predictions counted as incorrect.

MethodCorrect/scoredUnscoredAccuracy over scoredAccuracy over all 589
Full Context264/589044.82%44.82%
RAG286/589048.56%48.56%
Mem0293/589049.75%49.75%
A-Mem283/589048.05%48.05%
LightMem299/589050.76%50.76%
Mem0-g284/588148.30%48.22%
MemoryOS278/589047.20%47.20%
GAM290/589049.24%49.24%
MemAgent243/5286146.02%41.26%
Mem-α307/589052.12%52.12%
UMA269/589045.67%45.67%
MEM1230/5305943.40%39.05%
Memory-R2238/589040.41%40.41%
PersonaMap330/585456.41%56.03%

Mem0-g has one prediction without a parseable answer option. MemAgent has 59 questions affected by ingestion context-length overflow and two predictions without parseable options; MEM1 has 59 questions affected by the same context-length limit. PersonaMap has four empty predictions. All other listed methods score all 589 questions.

PersonaMap’s reported 56.41% corresponds to 330/585 scored predictions. Counting the four empty predictions as incorrect gives 56.03% (330/589). The final column provides a common-denominator comparison over the complete question set.

Responses to the remaining concerns continue in Part 2/2.

AuthorsOpenReview ↗

Response to reviewer yG5U

This continues our response to Reviewer yG5U from Part 1/2.

2) Update magnitude and repeated runs (W2, Q2).

Starting from the same SFT checkpoint, GRPO improves accuracy from 48.58% to 49.53%, while RICO reaches 53.08%—a gain of 4.50 percentage points over SFT and 3.55 points over GRPO. Thus, the evaluated RICO run achieves additional gains beyond its SFT initialization.

We agree that these results do not separately isolate the effects of credit selection and update magnitude. Matched-update GRPO controls and multiple training seeds are needed to distinguish these effects and assess stability. Fig. 5 characterizes the selectivity and scale of RICO’s updates; the QA comparison provides the downstream performance evidence.

3) Credit interpretation and estimation noise (W3, Q3).

RICO targets the expected continuation return conditioned on an action under the rollout policy (Sec. 3.3). Subsequent actions are sampled as part of that continuation. A reward difference between two individual trajectories therefore does not prove that their first differing action alone caused the outcome.

Natural credit averages repeated observations of each compared action, while CF evaluates factual and alternative branches from the same preceding context. These controls improve comparability but do not eliminate finite-sample noise. We have not conducted large-sample credit re-estimation and therefore cannot establish sign or ranking stability under substantially more samples.

For free-form AddClaim actions, CF first uses an observed contrasting action after near-paraphrase filtering, or StopExtraction when no accepted alternative is available. Graph CF for ClassifyRelation requires an existing alternative; otherwise the context is skipped. These rules are clarified in Appendix B.3.

4) Relation-label quality and model dependence (W4, Q4).

Relation construction is a learned component of PersonaMap. During SFT, the shared policy learns relation decisions from GPT-5.1 demonstrations. During RL, the evidence-sufficiency reward evaluates downstream outcomes, while RICO assigns action-level credit to eligible ClassifyRelation decisions. This allows relation construction to be refined beyond teacher imitation using feedback on its downstream utility.

ConflictsUnresolved represents incompatible claims under overlapping conditions, whereas DistinctByQualifier distinguishes claims that remain compatible under different temporal or contextual qualifiers (Sec. 3.1). Appendix E provides a worked example illustrating this distinction.

Replay validation checks execution consistency, while downstream reward provides task-level feedback; neither constitutes an independent measurement of relation-label accuracy. We have not separately measured this accuracy for GPT-5.1 or the trained policy. GRPO and RICO start from the same GPT-5.1-supervised SFT checkpoint, allowing their performance to be compared under shared teacher supervision.

5) Navigation seeds and seed-only sufficiency (Q5).

Before navigation, the retriever scores claims by TF-IDF cosine similarity between the question and claim text, then greedily selects up to eight seed regions. Each region corresponds to either an individual claim or an unresolved conflict group. Starting from these regions, the policy can inspect claims, traverse neighborhoods, and open conflict groups within a bounded interaction budget.

During pipeline development, preliminary comparisons favored graph navigation over using the initial seeds alone, motivating this design. However, a verified seed-only ablation using the final checkpoint is not yet available. We will complete and verify this controlled comparison to quantify the additional contribution of navigation.

6) Scope of memory maintenance (W5).

Forgetting is an offline maintenance operation in our current implementation. The shared policy learns claim extraction, relation construction, and navigation, while a separate deterministic, query-independent operator prunes memory (Sec. 3.4; Appendix C).

We first complete QA evaluation using the unpruned memory, then remove 50% of stored claims and re-evaluate the same questions. The original graph-level analysis reports approximately 80% complete mapped relational-evidence retention on eligible questions. In the additional end-to-end evaluation on 422 questions from held-out personas 7–9, QA accuracy decreases from 53.08% (224/422) to 39.10% (165/422), retaining 73.7% of its unpruned value. These metrics respectively measure evidence preservation and downstream answer accuracy.

For an online deployment, a predefined storage budget could trigger pruning whenever memory reaches that budget, allowing the same maintenance operator to be applied periodically. This is a possible deployment strategy; our current experiments evaluate offline compression rather than longitudinal graph growth, accuracy, and maintenance cost.

Thanks for the response and the extra results. The re-scoring on the full 422 questions clears up the question-count issue. The rest of my concerns are still open, though. Without a GRPO run matched in update size, and with only one seed, I still can’t tell whether RICO helps because of better credit or just smaller updates. I’m also not convinced by StopExtraction as the counterfactual for AddClaim, since stopping early will almost always look worse. Label accuracy and the seed-only comparison are still missing. On pruning, the drop from 53.08% to 39.10% is large, and without a random or recency-based baseline it’s hard to say the structure-aware rule adds much. I’ll keep my score.

AuthorsOpenReview ↗

Response to reviewer yG5U

Thank you for the follow-up. We would like to clarify our pruning results.

The original analysis already compares structure-aware pruning with random and sparsity-only pruning at matched compression budgets, showing better preservation of complete mapped relational evidence (Appendix C). This provides empirical support for the evidence-preservation rationale.

After removing 50% of claims, PersonaMap retains 73.7% of its original QA accuracy. This is a substantial fraction of the original performance given that answering a question can require several claims together. Losing even one necessary claim can make the question unanswerable, even when the rest of the evidence remains. PersonaMap explicitly links related claims and uses those connections during pruning to prioritize keeping claims that may be needed together, including conflicting claims that remain unresolved. We are extending the QA evaluation to include random and recency-based pruning baselines, as suggested.

Reviewer_8gpZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a framework for managing agents' personalized memory using a relational structure. The main contributions of the paper are: (1) a stage-conditioned shared policy for constructing and navigating a personal memory graph defined as a set of claims and clear relationships. (2) a new Credit-Optimized RL model, named RICO, for refining the policy mentioned in point one. (3) an offline pruning method based on the weakness of the claims in the graph, i.e., pruning sparsely connected claims. The framework is evaluated on SubtleMemory (Personas 7–9) and zero-shot on PersonaMem18, achieving 50.81% overall accuracy on SubtleMemory and 56.41% on PersonaMem, improving over other approaches. The authors also present controlled ablations showing that SFT + RICO reaches 50.81% compared to 48.12% for SFT + GRPO under identical data budgets. Under structure-aware forgetting, the system retains complete mapped relational evidence for 80% of eligible test questions at a 50% claim compression budget

Critical strengths
  1. Learning a joint memory lifecycle (extraction, structuring, and navigation), rather than combining fixed rule-based graph builders with separate retrieval models, is an interesting and innovative idea.
  2. Maintaining unresolved conflict groups in the memory to prevent loss of historical detail or contradictory user preferences over time is a nice idea.
  3. Although RL is not my area of expertise. The RICO model seems to address a critical problem, i.e., reward conflation in trajectory-level RL, and solves it by isolating the causal impact of specific tool decisions through natural forks and counterfactual re-rollouts.
Critical weaknesses
  1. I have two main critics/questions to the overall evaluation. The authors provide so little information about the settings of the methods compared in the “Personalized long-term memory results” Table 1. For example, how is the RAG system set up? Are the other values taken from other papers? Given that you use only 2 personas from PERSONAMEM, how do you calculate the numbers? Also, the evaluation relies heavily on synthetic dataset generation; i.e., dataset generation relies on GPT-5.1, and RL reward calculation relies on a frozen Qwen3.8-27B judge.
  2. The paper does not present performance evaluation after pruning; such evaluation is important to understand whether pruning affects the overall memory system.
  3. The graph definition is too obscure; without an example (as additional material), it is hard to understand the graph structure, which is important for reproducing the work.
Questions for authors

Can you provide more details about the setup of the methods compared in Table 1? How is the RAG system set up? Are the other values taken from other papers? Given that you use only 2 personas from PERSONAMEM, how do you calculate the numbers?

1 comment
AuthorsOpenReview ↗

Response to reviewer 8gpZ

Thank you for the constructive feedback. We appreciate your recognition: jointly learning the memory lifecycle is an interesting and innovative idea; maintaining unresolved conflict groups is a nice idea; RICO addresses a critical problem—reward conflation in trajectory-level RL. We address your concerns below.

1) Evaluation setup, baseline implementations, and PersonaMem coverage (W1; Questions for Authors).

All scores in Table 1 come from our own evaluation runs, rather than being copied from prior papers, because differences in evaluation settings limit the direct comparability of published results. Appendices D.1–D.3 document the evaluation protocols, baseline configurations, answer-generation and scoring procedures.

For RAG, SubtleMemory uses session-level indexing and top-12 retrieval by cosine similarity between normalized, mean-pooled Qwen2.5-7B-Instruct representations. PersonaMem uses approximately 700-token dialogue blocks and top-10 retrieval with all-MiniLM-L6-v2. Each block is encoded once, with right truncation to 256 WordPiece tokens and no sub-block aggregation; the indexed block size therefore differs from the encoder input window. Appendix D.2 provides these details and the construction models, embeddings, chunking, and retrieval budgets of the other baselines.

Regarding the “two personas” concern, the PersonaMem v1 32K evaluation set contains 20 personas and 589 questions. Under our persona-full protocol, we merge the longest available prefix for each shared context associated with a persona, construct its memory before revealing evaluation questions, and then answer its multiple-choice questions. This differs from the original question-specific end-index protocol, as stated in Appendix D.1. Overall accuracy is computed at the question level, rather than averaged across personas.

2) Teacher-generated demonstrations and the reward judge (W1).

GPT-5.1 generates training tool demonstrations from training histories; it does not generate our held-out evaluation questions or reference answers. The demonstrations are replayed through the executor to check tool execution validity. Their construction and validation are described in Appendix A.

The frozen Qwen3.8-27B judge supplies the evidence-sufficiency reward during RL training. Its prompt and decision rule are provided in Appendix B.3. This reward is separate from final benchmark scoring: SubtleMemory follows the benchmark’s evaluation protocol, using Gemini-2.5-Flash when LLM judgment is required, with majority voting over three calls; PersonaMem uses exact reference-option matching without an LLM judge (Appendix D.3).

These procedures distinguish teacher supervision, training rewards, and evaluation metrics.

3) End-to-end QA after pruning (W2).

We additionally evaluated downstream QA after pruning 50% of stored claims. On the same 422 questions from held-out SubtleMemory personas 7–9, accuracy decreases from 53.08% (224/422) to 39.10% (165/422), a reduction of 13.98 percentage points. With half of the claims removed, QA accuracy remains at 73.7% of its unpruned value.

The approximately 80% retention reported in the paper measures whether complete mapped relational evidence survives pruning on eligible questions; it is not answer accuracy. The additional QA evaluation directly measures the downstream performance cost of compression.

4) Graph definition and worked example (W3).

Section 3.1 defines entity nodes, atomic-claim nodes, conflict-group nodes, and typed semantic edges. Appendix E makes these definitions concrete through a worked example tracing source dialogue into claims and relations, followed by evidence retrieval and an answer (Figure 7: Case study of a cross-session conflict in alcohol preferences).

The example illustrates how complementary information is connected and conflicting alternatives remain accessible, linking the formal graph definition to the memory operations used by the system.

Reviewer_sagwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PersonaMap, a relational memory framework for personalized LLM agents, and RICO, an action-level RL algorithm. PersonaMap uses a single shared policy, initialized via SFT, to extract atomic claims from dialogues, build a relational memory graph, and traverse the graph to answer user queries. To refine the policy beyond trajectory-level GRPO, RICO assigns action-level credit using natural rollout forks and selective counterfactual rollouts, paired with an offline graph-pruning rule for memory forgetting.

Critical strengths
  • The proposed PersonaMap framework is clean and well-motivated, which formulates claim extraction, relational linking, and bounded graph traversal under a single shared policy. Maintaining explicit conflict groups instead of overwriting past claims is an effective way to handle evolving user preferences.
  • The proposed RICO combines execution-validated SFT with action-level credit assignment that addresses a real limitation of trajectory-level RL, where a single linking or retrieval mistake would otherwise penalize valid upstream memory operations.
Critical weaknesses
  • Grouping shared states across rollouts for step-level credit and running local rerollouts from intermediate states are already established in recent agentic RL works, including GiGPO (Feng et al., 2025), STeCa (Wang et al., 2025a), Memory-R2 (Yan et al., 2026a), and GraphGPO (Cheng et al., 2026).
  • RICO suffers from severe credit sparsity, as it requires an exact match of the preceding action prefix to form a natural fork. Because free-form claim extraction diverges early in action order, later graph-construction and navigation steps rarely merge even when reaching equivalent memory states.
  • The evaluation of the proposed PersonaMap and RICO is limited to two datasets, while standard benchmarks like LoCoMo and LongMemEval are omitted.

Cheng, Xin, et al. "Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning." ICML 2026.

Questions for authors
  1. When a rollout group has no natural credit across dozens of steps, how does RICO select which single decision context (BCF=1B_{CF}=1) to replay? When all K=16K=16 rollouts took the same action a at that context, how is the alternative action a' generated for open-ended actions like AddClaim vs. discrete actions like ClassifyRelation?
  2. What is the performance of SFT + RICO using only natural forks (BCF=0B_{CF}=0), and how does it compare to keeping a weighted trajectory-level GRPO advantage on unbranched steps (as in GraphGPO and Memory-R2)?
  3. How do Tool-Joint SFT and SFT + GRPO perform on PersonaMem, and what is the actual end-to-end QA accuracy on SubtleMemory (personas 7 to 9) after pruning 50% of claims?
  4. Please fix the figure typos and inconsistent notation. Figures 1 to 3 have frequent typos that resemble such as "Nature Credit", "Conterfactural Credit"(Fig. 1), "Seed Rigion" (Fig. 2), and "Boardcast" (Fig. 3). History and state notation shifts from (hkzh_k^z,SkzS_k^z) in Eq. (3) to an undefined context ss in Sec. 3.3 and hjh_j in Eq. (9).
2 comments
AuthorsOpenReview ↗

Response to Reviewer sagw

Thank you for the constructive feedback. We appreciate your recognition: the PersonaMap framework is clean and well-motivated; preserving unresolved conflicts effectively handles evolving user preferences; and RICO addresses a real limitation of trajectory-level RL. We address your concerns below.

1) RICO’s distinction from prior credit-assignment methods (W1).

We agree that shared-state grouping and intermediate rerollouts have established foundations. However, adapting to memory-tool policy remains challenging. Our contribution concerns their design for learning relational memory through individual tool decisions. Memory-R2 combines global optimization with local credit computed from session-level rerollouts starting from shared memory states. PersonaMap learns an explicit memory graph with typed semantic relations, while RICO estimates credit for individual memory-tool decisions.

RICO differs in how it estimates natural credit and invokes counterfactual evaluation.

(i) GiGPO compares returns within repeated-state groups. RICO matches the preceding action prefix, executor state, and policy-visible context, then aggregates terminal rewards by action and compares action-conditioned mean continuation values for alternatives with repeated support. GraphGPO instead derives advantages from graph distances in an aggregated state-transition graph.

(ii) RICO first reuses available natural credit and invokes budgeted counterfactual evaluation only when an entire rollout group provides no usable natural credit. These counterfactual comparisons estimate credit for individual actions, whereas STeCa uses step-level comparisons and reflection to construct calibrated trajectories.

2) Credit sparsity, counterfactual evaluation, and ablations (W2; Q1–Q2).

We acknowledge that strict context matching limits natural-credit coverage. Natural forks provide usable credit for 23.7% of questions; CF adds 9.5 percentage points, increasing coverage to 33.2%.

Context selection and alternatives (Q1). When an entire rollout group provides no usable natural credit, we select at most one unresolved shared-prefix context visited by at least two rollouts (BCF=1B_{\mathrm{CF}}=1). Candidates are ranked by action contrast (including discounted synthesized alternatives), then rollout support and earlier decision position, with deterministic tie-breaking.

For AddClaim, we use an existing contrasting action after near-paraphrase filtering, or StopExtraction if none is available. For ClassifyRelation, graph CF requires an accepted alternative already present in the sampled rollouts, such as a different relation label. Otherwise, the context is skipped without CF credit.

Ablations (Q2). The requested natural-only and trajectory-level-advantage ablations are still in progress. These require computationally expensive separate RL training runs, and completed performance comparisons are not yet available.

Responses to the remaining concerns continue in Part 2/2.

AuthorsOpenReview ↗

Response to Reviewer sagw

This continues our response to Reviewer sagw from Part 1/2.

3) Additional benchmark evaluation, PersonaMem ablations, and pruning QA (W3, Q3).

Additional benchmark evaluation (W3). We added LoCoMo, covering all ten conversations and 1,540 non-adversarial questions, without additional parameter training on LoCoMo. The shared evaluation framework is described in Sec. 4 and Appendix D.1; the LoCoMo-specific protocol, metrics, and exact answer/judge prompts are provided in Appendix F. All results below are from our own evaluation runs. F1 and the shared local judge score (J) are percentages.

MethodOverall F1Overall J
Full Context41.352.3
RAG30.537.5
Mem030.734.1
A-Mem30.539.2
LightMem21.221.2
Mem0-g15.315.3
MemoryOS28.133.5
GAM32.740.8
MemAgent12.613.3
Mem-α14.615.5
UMA19.921.8
MEM13.62.7
Memory-R2*42.449.9
PersonaMap35.343.1

The Memory-R2 checkpoint was trained on two LoCoMo conversations included in this evaluation.

PersonaMap achieves competitive performance under this unified evaluation protocol, supporting cross-benchmark generalization without LoCoMo-specific parameter training.

PersonaMem ablations (Q3). We additionally evaluated the requested variants on the same 589 PersonaMem-32K questions. Missing or invalid predictions are counted as incorrect.

MethodCorrect/totalAccuracy
Prompted Graph251/58942.61%
Tool-Joint SFT263/58944.65%
SFT + GRPO296/58950.25%
SFT + RICO330/58956.03%

RICO improves accuracy by 11.38 percentage points over SFT and 5.77 points over GRPO, providing additional cross-benchmark evidence for the learned memory policy. The RICO score uses all 589 questions as the denominator; the originally reported 56.41% corresponds to 330/585 valid predictions.

End-to-end QA after pruning (Q3). On the same 422 questions from SubtleMemory personas 7–9, pruning 50% of stored claims reduces QA accuracy from 53.08% (224/422) to 39.10% (165/422), a decrease of 13.98 percentage points. This complements the relational-evidence retention analysis with a direct downstream QA evaluation.

4) Presentation and notation (Q4).

Thank you for your careful reading. We have corrected the typos in Figs. 1–3 and clarified and standardized the notation for states, histories, and decision contexts throughout the manuscript.

Comments on the paper

2 comments
Area_Chair_bqRZOpenReview ↗

Reviewers: Please Respond to the Author's Rebuttal

Dear Reviewers,

The authors have thoughtfully responded to your reviews. Please take a minute to engage with their response and discuss whether it changes your opinions.

Thank you for your time and effort!

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear reviewers,

We notice that the rebuttal ddl is approaching to an end. We hope that our response has addressed your concern and we are happy to clarify any remaining questions you might have! Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.