acceptodds
Withdrawn from ICLR 2027

Self-Evolving LLM Memory Extraction Across Heterogeneous Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM-based assistants increasingly rely on memory extracted from past conversations to serve users over time. However, real-world assistants face diverse tasks requiring different types of memory, which existing work largely ignores. We formalize the heterogeneous memory extraction task and introduce BEHEMOTH, a benchmark that repurposes 18 existing datasets spanning personalization, problem-solving, and agentic tasks, using a downstream utility-driven metric for systematic evaluation. Our empirical analysis confirms that no single static extraction prompt is optimal for all task categories, and that existing self-evolving prompt optimization frameworks degrade when training tasks are heterogeneous. To address this, we propose CluE, a self-evolving method that restructures analysis hierarchically: training examples are partitioned into clusters, each cluster is analyzed independently to keep feedback concentrated, and cross-cluster insights are synthesized into a single prompt update. Experiments on BEHEMOTH show that CluE consistently outperforms prior self-evolving frameworks. Together, the task formulation, benchmark, and method provide a foundation for memory extraction across heterogeneous real-world tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Memory, LLM Memory, Self-Evolving Agents, Benchmark, LLM-based Agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fyEjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies memory extraction for assistants serving heterogeneous tasks. It builds the BEHEMOTH benchmark from 18 existing datasets and proposes CluE, which clusters training examples, analyses each cluster separately and merges the insights into a single extraction prompt.

Critical strengths

(1) Clear problem definition.

Scoring extraction by downstream utility makes different extraction prompts directly comparable.

(2) Empirical finding.

The static-prompt study shows that no single extraction prompt adapts all task types.

Critical weaknesses

(1) Limited novelty.

The central mechanism of CluE is an established prompt-optimisation technique applied to a new task. For example, UniPrompt (Juneja et al., Findings of ACL 2025) already clusters training data, collects feedback per cluster and merges it into a single prompt; the paper neither cites nor compares with it.

(2) Insufficient experiments.

The evaluation disregards the several important baselines, its advantage cannot stand out. For example, simply choosing the best existing static prompt for each task category already outperforms CluE (Table 1 vs. Table 2), yet no routing baseline is evaluated.

(3) No adaptation experiments.

Despite the “self-evolving” framing, the prompt is optimised once offline and then frozen, and its ability to adapt is never tested. For example, no experiment introduces a new task category or changes the task mix after evolution; even the continual experiment in section 6.3 keeps the prompt fixed.

(4) Small sample and a single model.

With few training examples and one backbone model, gains of a few points carry little reference value. For example, evolution uses only 330 training examples, and all main results rely on Qwen3-32B as extractor, optimiser and answerer at once.

(5) Syntax errors.

Table 3 omits the standard deviation for LongMemEval, although the stated protocol repeats runs for test sets of this size. Table 5 gives no citation for MathEquationBalancer or AIME. Fig. 4 and the AlfWorld panel of Fig. 5 use truncated y-axes, which visually exaggerate small differences. The case study in Fig. 8 contradicts the evolved prompt in Fig. 19: the former credits CluE with keeping a preference mentioned only in passing, while the latter requires each preference to be supported by at least two conversation threads.

Questions for authors

(1) How does CluE compare with the highly-similar baseline UniPrompt under the same optimisation budget, and what does it add beyond cluster-wise feedback?

(2) Does CluE outperform a simple router that selects among the existing static prompts for each conversation?

(3) When a new task category appears after evolution, can the evolved prompt be updated without degrading earlier categories?

(4) How large is the variance of the overall gain across independent evolution runs with different random seeds?

Reviewer_9rrZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies memory extraction across heterogeneous tasks, covering personalization, problem-solving, and agentic interaction. It introduces BEHEMOTH, a benchmark constructed from 18 existing datasets, and evaluates extracted memories through their contribution to downstream task performance. The proposed method, CluE, groups training examples by extraction scenario, analyzes each group separately, and synthesizes the resulting feedback into a shared extraction prompt. CluE achieves the strongest overall results in the main experiments. The benchmark offers a useful perspective on memory extraction, although the evidence does not yet fully distinguish the contribution of semantic clustering from additional analysis and explicit guidance on prompt organization.

Critical strengths
  1. The paper addresses a practical problem. A general-purpose assistant needs to retain personal preferences, factual information, and procedural experience, each requiring different extraction decisions. Studying these requirements together is a meaningful extension beyond evaluating memory within a single application.
  2. The method is straightforward and has a simple deployment interface. Although optimization involves several components, the final system uses one shared extraction prompt, without requiring a separate task router or clustering model during inference.
  3. The main results show a useful improvement in aggregate performance. In Table 2, CluE reaches an overall macro accuracy of 50.01, compared with 46.00 for Simple and 47.52 for GEPA. It improves over Simple in all three task categories, suggesting that separate analysis of different extraction scenarios can reduce some cross-task trade-offs.
  4. The improvements are not confined to a minimal starting prompt. Starting from Survey, CluE increases overall macro accuracy from 47.69 to 51.06. Repeating optimization with alternative extraction models also preserves an overall advantage, providing evidence that the procedure is applicable beyond the main backend.
  5. The comparison with full context provides a useful practical result. CluE reduces the average downstream context from 2,851 to 252 tokens while achieving 50.01 macro accuracy, compared with 52.68 for full context. This is a favorable trade-off between context consumption and task performance.
Critical weaknesses
  1. The construction of source–target pairs leaves uncertainty about what the benchmark measures. Some pairs appear to involve near-equivalent tasks rather than transfer of reusable experience. In Figure 6, the source and target retain the same objects and environment description, with the main difference being a paraphrased instruction. Reusing concrete execution steps may therefore be sufficient. The paper also allows multiple examples to share a target query without clearly explaining whether the split prevents overlap at the underlying task level.
  2. The contribution of semantic clustering is not sufficiently isolated. Random grouping with four clusters already reaches 49.48 macro accuracy, close to CluE’s 50.01. Meanwhile, optimization calls increase from 30 for MemEvolve to 221 for CluE. These results provide stronger evidence for dividing analysis into multiple groups than for the necessity of adaptive semantic clustering.
  3. The explanation based on the emergence of a structured memory taxonomy is confounded by the Proposer instructions. Figure 18 explicitly requests a taxonomy, category-specific rules, and conflict resolution through category boundaries. Consequently, the structure of the final prompt cannot itself establish that clustering discovered a better organization of extraction knowledge. A comparison using the same synthesis instructions without semantic clustering would be more informative.
  4. Some evolved rules have unclear or overly restrictive applicability conditions. Figure 19 requires preference memories to be supported by multiple conversation threads and temporal anchors, potentially excluding a useful preference stated clearly for the first time. It also refers to an undefined three-phase problem-solving framework. These instructions raise questions about whether the final prompt contains coherent, executable principles and whether its stated rules explain the observed behavior.
  5. The generalization claims extend beyond the demonstrated results. The strongest held-out advantage is on LongMemEval; CluE is below GEPA on GPQA-Diamond and does not surpass ACE on ToolBench. With the Gemini extraction backend, agentic macro accuracy falls from 38.46 for Simple to 35.85 for CluE. Moreover, the backend experiments repeat optimization while keeping the downstream answering model fixed, so they do not establish that a learned extraction policy transfers across memory consumers.
  6. The continual experiments provide limited evidence for the heterogeneous setting motivating the paper. They evaluate two tasks separately, both selected because CluE already outperformed MemEvolve in the single-step setting. This does not establish that the advantage persists when different types of tasks and memories share one evolving stream. Differences in the example pools and demonstration settings also limit direct comparisons between single-step and continual performance.
Questions for authors
  1. Appendix A.1 states that multiple examples may share the same target query. Is the training–test split performed over constructed triplets or over underlying task instances? Does it exclude shared target queries, source–target overlap across splits, and near-duplicate formulations of the same task? Given the close correspondence illustrated in Figure 6, how much of the benchmark requires transferable experience rather than reuse of a nearly matching trajectory?
  2. Section 3.2 states that Qwen3-32B is used to complete source conversations for all datasets, whereas Appendix A.1 says that personalization datasets directly retain their provided conversations and target queries. Were assistant responses in the personalization datasets regenerated, or were the original conversations preserved? These descriptions should be reconciled because they imply different dependencies between benchmark construction and the generation model.
  3. What evidence identifies semantic coherence as the source of CluE’s advantage over random grouping? Figure 4 shows that random grouping with four clusters recovers much of the improvement, while Table 10 documents substantially more optimization calls than MemEvolve. How does the comparison change when analysis resources, sample coverage, and the final synthesis instructions are held comparable? In particular, would an unclustered baseline given Figure 18’s explicit taxonomy and conflict-resolution instructions obtain similar benefits?
  4. Figure 19 instructs the extraction model to map memories to a three-phase problem-solving framework, but that framework is not defined in the prompt or explained in the method. What are the three phases, and what information allows the extraction model to apply them consistently? Is this an intended component of the extraction policy or an unresolved reference introduced during prompt evolution?
  5. The same prompt requires preference memories to appear in at least two distinct conversation threads and to have multiple temporal or event anchors. How should these requirements be applied when the input contains only one thread, or when a user explicitly states a durable preference for the first time? Are these requirements enforced in practice? If they are, how are valid first-time preferences retained; if they are not, how should the final prompt be interpreted as an explanation of the method’s behavior?
  6. How does the Analyzer distinguish an extraction failure from a downstream reasoning or execution failure? A successful answer does not necessarily imply that the memory contributed, and an unsuccessful answer does not necessarily imply that the memory was inadequate. Does the analysis use performance without memory to assess its contribution, or otherwise prevent downstream model limitations from producing inappropriate extraction rules?
  7. Appendix B.1 concludes that faithful compression alone does not produce useful memory. However, Table 6 reports 46.84 macro accuracy for Summary, compared with 37.84 without memory and 46.00 for Simple. These results show that summarization provides useful information, although it underperforms CluE. What supports the stronger conclusion? The discussion should also distinguish the effect of extraction instructions from the effect of optimizing those instructions against downstream feedback.
  8. The alternative-backend experiments rerun prompt evolution while retaining Qwen3-32B as the downstream answering model. Can the authors clarify whether the intended generalization claim concerns the optimization procedure, the learned prompt, or the extracted memories? The current evidence establishes that the procedure can improve overall performance with different extraction backends, but leaves transfer to different memory consumers unresolved.
  9. Would the continual advantage persist when personalization, reasoning, and agentic tasks alternate within one shared memory stream? Appendix A.3 also states that AlfWorld uses a demonstration in source trajectories but removes it when answering single-step target queries. How much of the memory benefit comes from retaining information supplied by that demonstration, rather than extracting new experience from the trajectory? This distinction matters when interpreting both the no-memory comparison and transfer between evaluation settings.
  10. Two implementation details also need clarification. Appendix A.2 specifies three new candidate prompts per round, whereas Figure 18 returns exactly one candidate object: is the Proposer invoked three times, and do those calls share the same analysis? Appendix B.2 further treats a limit of five memory items as a bound on memory length. Since item count does not bound token count, is there a separate output-length constraint, and is it applied consistently across methods?
Reviewer_ypejOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces BEHEMOTH, a benchmark for memory extraction across 18 datasets spanning personalization, problem-solving, and agentic tasks. It proposes CluE, which clusters training examples by extraction scenario, analyzes each cluster separately, and combines the feedback to improve a shared extraction prompt. CluE achieves stronger aggregate performance than the evaluated prompt evolution baselines.

Critical strengths
  1. Evaluating memory extraction through downstream task performance provides a practical measure of memory utility. Combining three task categories under a common protocol also enables the study of cross-task trade-offs.

  2. CluE improves overall macro accuracy from 46.00% to 50.01% with the Simple initialization and from 47.69% to 51.06% with Survey. Experiments with additional extraction models further support its practical usefulness.

Critical weaknesses
  1. Data split independence is insufficiently specified. Appendix A.1 explicitly allows multiple examples to share a target query, but does not clarify whether these examples are grouped into the same split. Since optimization observes target conversations and rewards, overlap could weaken the generalization claim. The paper should report task-level overlap and the number of unique targets. This is an unresolved concern, not confirmed leakage.

  2. The benchmark construction provides limited evidence of realistic memory reuse. BEHEMOTH repurposes existing datasets rather than collecting interactions with naturally occurring dependencies between past conversations and future requests. In particular, problem-solving and agentic examples are constructed by generating a trajectory and retrieving a semantically similar task as the target.

  3. Limited coverage of memory baselines. The comparison focuses primarily on prompt optimization frameworks rather than a sufficiently broad range of memory methods published in 2025 and 2026. The related work discusses approaches such as ALMA and MemSkill, but the experiments do not include their relevant mechanisms or comparable extraction-focused adaptations.

Questions for authors
  1. Are original task identities and shared target queries disjoint across training and testing? Please clarify the split construction and provide overlap statistics.

  2. Does CluE reliably outperform random grouping across independent evolution runs under matched optimization budgets?

Comments on the paper

1 comment

We thank the reviewers for their comments. Our work addresses memory extraction across heterogeneous tasks, and self-evolving optimization under heterogeneous training distributions, where existing frameworks tend to trade off performance across task types. We briefly respond to the main concerns below.

Regarding UniPrompt, while both methods group examples before analysis, UniPrompt refines a single-task prompt using one cluster at a time, whereas CluE combines insights from all clusters into each update. This distinction matters when one prompt must serve heterogeneous tasks and is optimized only indirectly through downstream utility. We will include UniPrompt in the comparison in future versions.

Routing-based methods are complementary to our focus on prompt optimization. A routed prompt can serve as the starting point for evolution (Table 4), and a single learned prompt remains useful when one conversation contains several kinds of memory.

On data splits, 173 of 2,291 test pairs (7.6%) share a target instance with the optimization set. All methods share identical splits and settings, so the comparison remains fair, and CluE also achieves the best result on LongMemEval, a held-out dataset never seen during evolution.

We also note that the evaluation is not limited to a single model. Evolution was repeated with Gemini-3-Flash and GPT-5.4-mini as extraction backends (Appendix B.3), and results on smaller test sets are averaged over three runs.

We thank the reviewers again for their time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.