acceptodds
Under review as a conference paper at ICLR 2027

From Experience to Policy: Structured Memory for Continual Learning in LLM Agents

Abstract

Large Language Model (LLM) agents increasingly rely on external memory for long-horizon reasoning. However, existing mechanisms primarily store passive interaction histories, which support retrieval but lack explicit reusable decision policies. Consequently, agents struggle to accumulate and refine experience across tasks. We propose Structured Policy Memory (SPM), a continual learning framework that transforms interaction trajectories into explicit policy knowledge organized as a structured graph. SPM establishes a closed-loop process where the agent retrieves relevant policies to guide decisions and validates newly extracted knowledge to update the memory after each interaction. Extensive experiments on LoCoMo, LongMemEval, StrategyQA, and ALFWorld demonstrate that SPM consistently outperforms representative prompting, retrieval, and memory-augmented baselines while maintaining computational efficiency. Specifically, SPM achieves 33.25% overall Semantic Similarity on LoCoMo (the best among all compared methods), exhibits the largest continual learning gain of +13.17pp across 1500 tasks, and attains 95.78% Action Match on ALFWorld. These results demonstrate effective experience reuse and sustained performance improvement, providing a practical foundation for continual learning in LLM agents. Our code is available at https://github.com/linda-2-hh/SPM.

Primary area:
interpretability and explainable AI
Keywords:
Large language model agents, continual learning, policy memory, experience reuse, embodied AI, reasoning agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nGmUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Structured Policy Memory (SPM), an external-memory framework that represents agent experience as a typed, weighted graph containing facts, states, actions, outcomes, rules, and constraints. Retrieved subgraphs guide answer generation, while knowledge extracted after each interaction is evaluated by a critic and incorporated into memory. Experiments cover LoCoMo, StrategyQA, LongMemEval, and ALFWorld. The paper presents these results as evidence for improved memory performance, continual policy learning, and favorable computational efficiency.

Critical strengths

The paper addresses a relevant problem in experience reuse. Converting accumulated interactions into reusable guidance for future decisions is an important objective for long-running LLM agents. A representation that combines factual information with behavioral constraints is a plausible direction, particularly when future queries require relationships between multiple pieces of experience.

The appendix provides useful visibility into the proposed system. The submission includes algorithmic pseudocode, extraction and reasoning prompts, graph-retrieval ablations, critic ablations, and failure examples. These materials help identify how memory updates operate and where the approach can fail. The authors also acknowledge ordering bias and the limitations of critic filtering, although these qualifications are not adequately reflected in the main conclusions.

Critical weaknesses

Essential experimental specifications are missing, and central claims conflict with the reported results.

Section 5.1 (p. 6, lines 306–311) states that all experiments use the same underlying LLM but does not identify its name, size, checkpoint, or version. Neither the main text nor the appendix explicitly identifies the actual backbone, embedding model, or model used for critic evaluation, and important generation settings are not specified. These omissions do not permit reproduction from the manuscript alone and make it difficult to assess baseline comparability or interpret the latency measurements. A statement that methods share a model does not replace identifying the model and explaining each method’s configuration.

There are also substantive claim–evidence conflicts. Table 6 (p. 14) reports SPM at 0.3325, below Mem0 at 0.3372 and KG-RAG at 0.3329, while its accompanying text explicitly claims superiority over these methods. Both baselines are absent from Table 1, which supports the headline claim of best overall LoCoMo performance. Table 8 (p. 15) reports ALFWorld Action Match of 95.78% for SPM, below BM25 (99.40%), Reflexion (98.20%), and A-Mem (99.80%). These results do not support the broad statements of consistent superiority. Additionally, Table 4 and Tables 10/14 all specify the first 100 LoCoMo questions, but report conflicting results: Table 4 gives 8.0% EM for both SPM and RAG (BM25), whereas Tables 10 and 14 consistently report 6.0% for SPM and 5.0% for RAG (BM25). KG-RAG likewise changes from 8.0% in Table 4 to 4.0% in both Tables 10 and 14. Table 1 labels the overall measure Semantic Similarity, whereas Table 6 reproduces matching values for many methods under “Success Rate.” The manuscript does not reconcile these differences.

The concern is not a requirement to outperform every baseline. The experimental conditions and reported outcomes are insufficiently specified and internally inconsistent, preventing a dependable assessment of the claimed contribution.

The evaluation does not establish continual improvement through policy evolution.

Section 5.3 and Table 7 evaluate cumulative scores over changing question prefixes. The fixed-index BM25 baseline improves by 10.50 percentage points, demonstrating a substantial question-composition effect. Subtracting this change from SPM’s 13.17-point gain does not identify a learning effect: different systems can respond differently to changing questions even without memory evolution. The within-conversation early-versus-late comparison in §5.7 has the same limitation. A same-system frozen-memory control or fixed held-out probes is needed to attribute improvement to the update mechanism.

Algorithm 1, line 10, and Table 11 (p. 17) explicitly provide the ground-truth answer to the extraction module after prediction. This can be a legitimate supervised online protocol, but equivalent feedback access across baselines is not established. The paper therefore does not adequately distinguish acquisition of reusable policies from accumulation of answer-derived factual information. This is not evidence that the current answer was exposed before prediction; it is a concern about information available for subsequent questions and the comparability of the evaluation.

The available mechanism ablations do not resolve the issue. Removing policy evolution reduces Semantic Similarity by only 0.19 points on the 500-question subset (Tables 2/9). Removing the critic improves EM from 17.60% to 18.00% and Semantic Similarity from 31.53 to 31.86 (Table 13, p. 21). The authors acknowledge this subset result and assert that filtering becomes beneficial at larger scales, but do not provide the corresponding full-length quantitative comparison. Consequently, the central claim of sustained improvement through critic-guided policy evolution remains unsubstantiated.

Questions for authors

Actual models and configurations: What exact backbone, embedding model, and critic model were used? Please provide checkpoint/API versions, decoding configurations, module assignments, and the corresponding baseline settings. Where are these specified in the submitted materials? How are graph construction, extraction, and critic calls included in the efficiency accounting?

Reconciliation of the experimental record: Please reconcile Tables 1 and 6, the stated superiority over Mem0/KG-RAG, and the first-100-question EM discrepancies between Tables 4 and 10/14. Are these different metrics, subsets, or protocols? Which results support the abstract and conclusion after reconciliation?

Feedback and information boundaries: Which baselines receive ground-truth answers after each query? When is memory reset, and how are repeated or closely related questions handled? Is the initial graph built from the same permitted corpus for every method? Please reconcile the empty-graph description in §5.1 with BuildInitialGraph(M) in Algorithm 1.

Controlled evidence for evolution and critic filtering: Can the authors compare the same SPM system with evolving memory, frozen memory, and no critic over the full evaluation length under identical feedback? Fixed held-out probes or multiple randomized question orders would help separate memory updates from question-composition effects. Please provide uncertainty across independent runs or orders.

Meaning of cross-benchmark results: Please define ALFWorld Action Match and the LongMemEval temporal reasoning score, including their denominators, aggregation, and evaluation procedures. Does ALFWorld involve closed-loop execution, and what is the episode-level task success rate? Do the additional benchmarks evaluate transfer of an existing policy memory or separately initialized systems? These distinctions materially affect the claimed generalization and decision-making conclusions.

1 comment
AuthorsOpenReview ↗

Response to Reviewer nGmU

Thank you for these questions. We address the reported results and the limits of the submitted record.

1. Models and efficiency. Section 5.1 states that methods share an underlying LLM, but the submission does not give its checkpoint or API version. It also does not identify the embedding checkpoint, a separate Critic model, decoding settings, module assignments, or complete baseline configurations. Algorithm 1 specifies the roles of answering, extraction, and CriticEvaluate, but not their exact implementations.

Table 4 reports 46,691 ms average latency and 535 input plus 24 output tokens per question for SPM on the first 100 LoCoMo questions. It does not itemize graph construction, retrieval, post-answer extraction, Critic calls, or graph updates. We therefore cannot establish from that table that the reported latency and tokens include the full cost of maintaining an evolving memory.

2. Experimental record. Appendix Table 6 gives the LoCoMo category values on a 0–1 scale and adds methods omitted from Table 1. Although its caption calls them “Success Rate,” the corresponding main-table values are labeled Semantic Similarity. SPM scores 0.3325 overall, below Mem0 (0.3372) and KG-RAG (0.3329). “Best overall” applies only to the methods displayed in Table 1; superiority over Mem0 and KG-RAG is not supported by Appendix Table 6.

Table 4 reports 8.0% EM for SPM on the first 100 LoCoMo questions, while Tables 10 and 14 report 6.0%. The submission documents no difference in subset, metric, or protocol that explains the two-question discrepancy. We cannot reconcile it by assuming a different evaluation run.

The supported conclusion is narrower than the abstract’s: SPM reaches 33.25% overall LoCoMo Semantic Similarity and performs competitively across the reported tasks. The complete set of reported comparisons does not establish best LoCoMo performance or consistent superiority across benchmarks.

3. Feedback and information boundaries. Algorithm 1 pairs each query with its ground-truth answer. SPM predicts first; extraction then receives the question, prediction, and ground truth. Accepted triples can enter memory and be retrieved for later questions. The submitted protocol does not establish which baselines receive equivalent post-prediction feedback. It also does not specify a reset between conversations or controls for repeated facts and closely related questions. Consequently, the observed learning curve may include benefits from earlier answer feedback and question ordering.

Section 5.1 describes an empty initial graph, whereas Algorithm 1 calls BuildInitialGraph(M). These descriptions agree if MM is empty at initialization, but the submission does not specify that condition or demonstrate that every method starts from the same permitted corpus. Equal information access and conversation isolation cannot be verified from the reported protocol.

4. Evolution and Critic controls. The 500-question ablation compares full SPM with policy evolution disabled: Semantic Similarity is 31.53 versus 31.34. A separate ablation finds that removing the Critic increases EM from 17.60% to 18.00% and Semantic Similarity from 31.53 to 31.86%. The submitted results do not compare evolving, frozen, and no-Critic memory over the full 1,500-question stream under identical feedback.

SPM’s score rises by 13.17 percentage points between 50 and 1,500 processed questions, but a fixed-index BM25 baseline also rises by 10.50 points. This indicates that question order or composition affects the trend. Without fixed held-out probes, randomized orders, and uncertainty across independent runs, the additional increase cannot be assigned confidently to graph evolution. The reported 500-question ablation does not show an accuracy benefit from Critic filtering.

5. Cross-benchmark interpretation. Figure 5 labels the ALFWorld evaluation n=100n=100, and Table 8 reports 95.78% Action Match. The submission does not specify whether its denominator counts actions, action sequences, or another unit, nor does it give the matching procedure or episode-level success rate. Action Match does not establish closed-loop task completion.

Figure 5 labels LongMemEval n=133n=133 and reports a temporal reasoning score of 5.26, but does not sufficiently define its denominator, aggregation, or scoring procedure. The submission also does not state whether these benchmarks reuse a policy memory learned elsewhere or initialize separate systems. Their results therefore cannot establish cross-benchmark transfer of an existing memory or successful embodied decision making.

Reviewer_h3YdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper propose Structured Policy Memory (SPM) to represent agent experience as a typed, weighted graph containing states, actions, outcomes, and other factors. For each query, it retrieves similar nodes, expands a local subgraph, and supplies the serialized graph to an LLM. After answering, it extracts candidate knowledge and filters additions through a critic scoring consistency, utility, and novelty.

Critical strengths

Experimental results provide preliminary evidence that structured memory can improve answer generation. On the 500-question LoCoMo subset, removing graph retrieval reduces semantic similarity from 31.53 to 27.97, while removing the graph representation reduces it to 28.19. Both changes also reduce the other reported generation metrics. Those results support the value of typed graph representations and graph expansion.

Critical weaknesses

While the authors claim that continual policy improvement is the primary contribution, the direct evidence is weak. In the experiments, disabling the evolution changes semantic similarity by only 0.19 points on 500 questions. It is hard to make readers believe that that evolution becomes more beneficial over longer horizons.

From the experiments, it is hard to judge whether SPM outperforms existing methods and what its reported performance actually demonstrates.

  • LoCoMo: The abstract describes 33.25% as best among all methods, but Appendix Table 6 reports 33.72% for Mem0 and 33.29% for KG-RAG, both above SPM’s 33.25%. These baselines are absent from the main LoCoMo table.
  • ALFWorld: Appendix Table 8 reports 99.80% for A-Mem, 99.40% for BM25, and 98.20% for Reflexion, compared with 95.78% for SPM. Thus, the reported action-match result does not establish comparative superiority.
  • Inconsistent evaluation values: On the “first 100” LoCoMo questions, Table 4 reports SPM at 8% EM and BM25 at 8%, whereas Tables 10 and 14 report 6% and 5%, respectively. Different runs or settings could explain this, but none are identified.
Questions for authors

Does every evaluated method receive each question’s correct answer after prediction? Can that answer be stored and retrieved for later questions? How are related questions, repeated facts, and conversation boundaries handled?

Are Appendix Tables 6 and 8 directly comparable with the main results? If so, how should the claims of best LoCoMo performance and consistent cross-benchmark superiority be revised? What explains the differing first-100-question EM results?

2 comments
AuthorsOpenReview ↗

Response to Reviewer h3Yd

Thank you for pointing out these issues. They affect both the interpretation of continual learning and the scope of our performance claims.

1. Answer feedback and memory boundaries. In Algorithm 1, each query is paired with its ground-truth answer, gt_i. SPM predicts first, then passes the question, prediction, and gt_i to the extraction step. Accepted knowledge is added to the graph and can be retrieved for later questions. Thus, SPM receives post-prediction answer feedback. The reported improvement over a question stream cannot be attributed solely to learning from the agent’s unaided experience.

The manuscript does not establish that every evaluated baseline receives the same feedback or has the same opportunity to retain it. It also does not specify controls for related questions and repeated facts. A later question may therefore benefit from information supplied with an earlier answer. Although LoCoMo has conversation identifiers, Algorithm 1 does not define a graph reset or retrieval boundary between conversations. The reported continual-learning curve should be read as performance under this answer-feedback protocol, with possible carryover between related questions; it is not evidence of transfer across isolated conversations.

2. Appendix Tables 6 and 8. Appendix Table 6 reports the same type of category scores as main Table 1, so its additional baselines matter to the ranking claim. Its caption calls the values “Success Rate,” but the values are presented as Semantic Similarity in the main comparison. SPM scores 0.3325 overall, below Mem0 (0.3372) and KG-RAG (0.3329). Accordingly, the claim that SPM has the best overall LoCoMo performance holds only among the methods displayed in Table 1, which omits those two comparators. It does not hold across all methods reported in the paper.

Appendix Table 8 also narrows the cross-benchmark claim. SPM ties A-Mem on StrategyQA EM at 62.00%; on ALFWorld Action Match, SPM scores 95.78%, below A-Mem (99.80%), RAG-BM25 (99.40%), and Reflexion (98.20%). The results support competitive performance on these measures, not consistent superiority across benchmarks. Action Match should also not be interpreted as ALFWorld task success.

Finally, the two reported first-100-question LoCoMo EM values for SPM conflict: Table 4 gives 8.0%, whereas Appendix Tables 10 and 14 give 6.0%. All describe the first 100 questions, and the manuscript supplies no difference in sample selection, scoring, or run configuration that explains the two-question discrepancy. We cannot treat these as directly comparable estimates or claim an established explanation from the submitted record.

Thank the authors for your responses and the additional experimental results, which partially addressed my main concerns. However, I would keep my original rating.

Reviewer_voLKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Structured Policy Memory (SPM) organizes interaction experience into a graph containing facts, states, actions, outcomes, and constraints. The agent retrieves a subgraph to guide answering, then extracts new knowledge using feedback and updates the graph after Critic filtering. The paper evaluates memory performance and continual improvement on LoCoMo, StrategyQA, LongMemEval, and ALFWorld.

Critical strengths

The explicit graph makes stored knowledge and constraints inspectable. The graph-structure and traversal ablations also provide some evidence that organizing related information helps downstream answering.

Critical weaknesses
  1. The contribution beyond existing graph memory is unclear. The method combines typed nodes, similarity-based seed selection, graph traversal, prompted knowledge extraction, and a fixed scoring rule. It is not clear what makes the resulting memory a learned policy rather than a knowledge graph containing facts and textual lessons. The benefit of policy-specific representations should be isolated under matched memory content and feedback.

  2. The Critic is not sufficiently justified. Consistency, support, utility, and novelty are combined through manually chosen weights, but the computation and validation of these scores—particularly future utility—are insufficiently explained. Table 13 further shows that removing the Critic improves EM from 17.60% to 18.00% on 500 questions, so the reported ablation does not establish its necessity.

  3. The continual-learning evaluation is confounded by question order. The cumulative score is measured over changing sets of questions. Even fixed-index RAG improves by 10.50 points, compared with SPM’s 13.17 points. Subtracting these gains does not adequately control for differences in question difficulty or method-specific performance across question types.

  4. Retrieval quality as the graph grows is not evaluated. Algorithm 1 uses ten similar seed nodes and two-hop traversal, but does not adequately explain how irrelevant neighbors, conflicting knowledge, and expanding subgraphs are controlled. The paper should report retrieval quality, answer accuracy, and latency against graph size under a fixed retrieval budget.

Questions for authors
  1. What distinguishes SPM’s reusable policies from facts and textual lessons stored in an ordinary graph memory?

  2. How are the four Critic scores computed and validated, given that removing the Critic improves the reported 500-question results?

  3. How do retrieval quality, answer accuracy, and latency change as the graph grows under a fixed retrieval budget?

1 comment
AuthorsOpenReview ↗

Response to Reviewer voLK

Thank you for these questions. We clarify the representation and distinguish the reported evidence from claims that the experiments do not establish.

1. What distinguishes SPM’s policies from facts and textual lessons?

SPM represents decision-oriented knowledge connecting an applicable task or state to an action, outcome, or behavioral constraint. Its graph contains tasks, states, actions, outcomes, rules, and constraints; relations encode transitions, support, and restrictions. Retrieval uses the query and current state to return a subgraph that guides subsequent decisions.

A fact records an observation, such as an object’s location. A policy expresses how information should guide behavior under particular conditions. A textual lesson can express the same principle, but SPM makes its conditions, actions, and supporting relations separately accessible through graph retrieval and incremental updates.

We do not claim that ordinary graph memories cannot represent policies, or that every SPM triple is itself a policy. Factual nodes also support decisions. The contribution is the integration of trajectory extraction, decision-oriented graph organization, retrieval, and feedback-driven updates. A graph memory implementing these functions would substantially overlap with SPM.

Removing graph structure reduces Semantic Similarity from 31.53 to 28.19 on 500 questions. This supports the usefulness of the structured representation, but does not establish that every retained rule transfers across tasks.

2. How are the Critic scores computed and validated?

The four scores assess:

  • Consistency: Compatibility with existing graph knowledge.
  • Support: Evidence in the source trajectory.
  • Utility: Expected usefulness for future decisions.
  • Novelty: Additional information relative to existing memory. Appendix D identifies embedding cosine distance as the basis of this estimator.

The aggregate score is:

S(c)=0.30Scons(c)+0.25Ssup(c)+0.25Sutil(c)+0.20Snov(c).S(c)=0.30S_{cons}(c)+0.25S_{sup}(c)+0.25S_{util}(c)+0.20S_{nov}(c).

Candidates exceeding 0.55 are retained. The submission does not fully specify numerical rubrics for consistency, support, and utility, or validate the scores against independently annotated candidate quality. These are heuristic assessments, not calibrated probabilities of correctness or usefulness.

Validation consists of downstream sensitivity experiments and ablations. When one weight varies, the others are proportionally rescaled. The consistency and novelty sweeps yield EM between 17.20% and 18.00% on 500 questions. This shows limited downstream sensitivity in that setting; it does not independently validate each score.

The reviewer is correct about the Critic ablation. “No Critic” achieves 18.00% EM and 31.86 Semantic Similarity, versus 17.60% and 31.53 for the full configuration, and improves the other listed generation metrics. The EM difference is two questions. Without uncertainty estimates, neither direction supports a reliable performance advantage.

Thus, the 500-question experiment does not demonstrate an accuracy benefit from filtering. Over-filtering useful candidates is a plausible explanation, particularly for similarity-based novelty, but no candidate-level audit establishes that mechanism. The claim that filtering becomes beneficial at larger scales is also not supported by a reported matched Critic-versus-no-Critic scaling experiment.

3. How does performance change as the graph grows under a fixed retrieval budget?

The submission does not report a controlled graph-growth experiment that jointly measures retrieval quality, answer accuracy, and latency under a fixed retrieval budget.

The available results measure separate aspects:

  • Continual learning: Semantic Similarity improves by 13.17 percentage points between 50 and 1,500 processed tasks. Experience, graph contents, and evaluated questions change together, so this does not isolate graph size.
  • Retrieval: On the first 100 LoCoMo questions, SPM reports P@1 = 0.060, R@5 = 0.159, MRR = 0.309, and answer EM = 6.0%. Passage-matching metrics may undercount useful abstracted policies, but low scores can also indicate missing evidence. Answer EM alone cannot distinguish these explanations.
  • Latency: The separate efficiency evaluation reports approximately 46.7 seconds per question. It does not provide measurements by graph size or separate retrieval time from generation time.

Consequently, the reported experiments do not establish the relationship between graph growth and these three outcomes under a fixed budget. Bounding the returned context limits prompt size, but does not itself bound graph-search cost or ensure that relevant policies remain retrievable as competing candidates accumulate. These scaling properties remain unmeasured.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.