acceptodds
Under review as a conference paper at ICLR 2027

Choosing How to Remember: Adaptive Memory Structures for LLM Agents

Abstract

Memory is critical for enabling large language model (LLM)–based agents to maintain coherent behavior over long-horizon interactions. However, existing agent memory systems suffer from two key gaps: they rely on a one-size-fits-all memory structure and do not model memory structure selection as a context-adaptive decision, limiting their ability to handle heterogeneous interaction patterns and resulting in suboptimal performance. We propose a unified framework, FluxMem, that enables adaptive memory organization for LLM agents. Our framework equips agents with multiple complementary memory structures. It explicitly learns to select among these structures based on interaction-level features, using offline supervision derived from downstream response quality and memory utilization. To support robust long-horizon memory evolution, we further introduce a three-level memory hierarchy and a Beta Mixture Model–based probabilistic gate for distribution-aware memory fusion, replacing brittle similarity thresholds. Experiments on two long-horizon benchmarks, PERSONAMEM and LoCoMo, demonstrate that our method achieves average improvements of 9.18% and 6.14%.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Large Language Models, Agent Memory, Long-Horizon Interaction, Memory Organization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_focyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Previous:

FluxMem studies memory-structure selection for agents: different query types favor different memory organizations (flat, episodic, semantic), and a learned selector picks the structure per query. Experiments across dialogue-memory benchmarks and backbones report gains over fixed structures. However, no oracle or fixed-structure upper-bound control is provided, the selector's own accuracy is never reported, and some baselines (ZEP 7.92, Mem0 27.35) are far below publicly reported numbers.

Updated:

FluxMem selects linear, graph, or hierarchical organization for memory units during updatint the memory. It combines this with a three-level memory and fusion operations. The proposed method is evaluated on benchmarks like personamem and LoCoMo. Therefore, some of my orginal review about per-query selection was incorrect.

Critical strengths

Previous:

Valuable perspective:Treating memory structure as a learnable, per-query choice is a sensible extension of the memory-system design space. Complete implementation and coverage:A full pipeline (selector + multi-structure store) evaluated across benchmarks and backbones.

Updated:

The added controls provide clear evidence for learned selection. For example, the proposed method achieves 51.20 F1, versus 46.12 of the state-of-the art method. The selector itself is cheap without additional LLM calls.

Critical weaknesses

Selector contribution never isolated:Without oracle or fixed-structure controls, gains cannot be attributed to selection itself; selector accuracy is unreported. Abnormal baselines, insufficient ablations:ZEP/Mem0 baselines are far below public reproductions; missing ablations and significance tests weaken credibility.

Update:

The selector in FluexMem report 0.75 accuracy with only four test decisions. This result is still limited for the generalization. Besides, the cross dataset results are useful,while the robustness to changes in future query-type proportions is not measured.

Questions for authors
  1. What is the selector's accuracy and the oracle upper bound? How much gain survives selection errors?
  2. Why are the ZEP/Mem0 baselines far below published numbers (version, config, protocol)?
  3. What overhead does the selector add (LLM calls, latency)?
  4. Is the selection policy stable under shifts in the query-type distribution?
Reviewer_oCvgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents FluxMem, which investigates the adaptive selection of memory organization structures for long-term dialogue. The system adopts a three-level memory architecture consisting of short-term interaction memory, mid-term episodic memory, and long-term semantic memory. For each mid-term memory unit, the system selects one of three organizational structures: linear, graph-based, or hierarchical. The structure selector is a lightweight MLP trained using offline supervision constructed from answer quality and memory utilization effectiveness. In addition, the paper proposes a fusion gate based on a Beta Mixture Model (BMM), which uses the distribution of candidate matching scores to determine whether two memory units should be merged.

Critical strengths
  1. Treating the memory organization structure itself as a selectable variable, rather than always adopting a fixed organization scheme, is a valuable direction for handling heterogeneous dialogue content.
  2. The combination of a lightweight structure selector with hierarchical memory is a reasonable attempt to balance organizational flexibility, retrieval effectiveness, and memory maintenance cost.
Critical weaknesses
  1. The paper assigns one structure to each mid-term memory unit. However, the same dialogue segment may simultaneously contain temporal information, entity relationships, and topic hierarchies, and these different aspects may be useful for different downstream queries. The paper does not sufficiently justify the advantages or disadvantages of selecting a single structure compared with maintaining a multi-view representation. More importantly, it is unclear how structure conflicts are handled during memory fusion. For example, suppose the current memory is predicted to be best represented by a hierarchical structure, while its candidate merge target is already organized as a graph. Which structure determines the organization of the merged memory? Is one structure converted into the other, is one representation discarded, or is a new structure selected after merging? This issue seems important because the proposed framework explicitly treats memory structure as an adaptive decision.
  2. The paper uses a matching function score(m,c), but it is not sufficiently clear what representations are used as its inputs, how the score is computed, or whether the matching function depends on the underlying memory structure. Moreover, although the w/o BMM ablation in Tables 3 and 4 provides some evidence for the usefulness of BMM, the alternative fusion strategy and its threshold-selection protocol are not described clearly enough. In particular, it is difficult to determine whether BMM actually outperforms a reasonably tuned simple baseline such as: Merge the candidate memory if its highest matching score exceeds a fixed threshold; otherwise, create a new memory. A controlled comparison against such a baseline, including how the threshold is selected, would make the benefit of BMM much more convincing.
  3. The motivation for BMM is based on the observation that matching-score distributions vary across different dialogue types and stages. However, a high-mean component in the score distribution only indicates relatively high matching scores; it does not necessarily imply that the corresponding candidates are actually suitable for merging. The paper does not directly show the matching-score distributions under different scenarios, nor does it establish a clear correspondence between the high-score component identified by BMM and the ground-truth merge decisions. This raises a more fundamental concern: if two different scenarios produce exactly the same set of matching scores but require different merge decisions, the proposed BMM gate has no information with which to distinguish them. Therefore, the paper needs to demonstrate that scenario-dependent differences in the appropriate merging criterion can indeed be identified from the matching-score distribution itself.
Questions for authors
  1. The paper selects a single structure for each memory unit. In practice, however, a memory may simultaneously benefit from multiple organizational structures—for example, a hierarchical structure for topics and a graph structure for entity relationships. Furthermore, during fusion, the current memory may prefer a hierarchical structure while the candidate memory is represented as a graph. Does the proposed framework have a mechanism for resolving such structural conflicts? If so, how is the structure of the merged memory determined? More generally, why is a single-view structure preferable to a multi-view representation in this setting?
  2. How exactly is the matching score computed? What are the input representations and the scoring function, and does the score depend on the memory structure? More importantly, is the proposed BMM-based fusion strategy demonstrably better than a simple fixed-threshold baseline that merges the candidate with the highest score when that score exceeds a tuned minimum threshold, and otherwise creates a new memory?The paper argues that "different scenarios may require different merging criteria." However, the proposed gate appears to operate only on the matching-score distribution and does not explicitly observe the scenario or dialogue context. Can the authors establish a concrete link showing that scenario-dependent differences in the optimal merging criterion are actually identifiable from the score distribution? Ideally, the paper should provide score distributions for different scenarios together with ground-truth merge labels, and demonstrate that the BMM components correspond meaningfully to mergeable versus non-mergeable candidates.
Reviewer_hNRrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FluxMem, an adaptive long-term memory framework for LLM agents that integrates linear, graph-based, and hierarchical memory structures. It employs a learned selector to determine appropriate memory structures based on conversational features and incorporates a Beta Mixture Model (BMM) for adaptive memory fusion. Experimental results on PERSONAMEM and LoCoMo show that FluxMem outperforms existing memory baselines.

Critical strengths
  • Strong empirical performance. FluxMem achieves competitive results on two long-term memory benchmarks, with notable improvements on PERSONAMEM.

  • Comprehensive framework. The method integrates multiple memory structures, learned selection, and probabilistic memory fusion into a unified system.

Critical weaknesses
  • Multi-structure memory organization, adaptive routing, and hierarchical management have been explored in prior work. The proposed framework largely integrates existing techniques, offering limited novelty.

  • The ablations do not directly isolate the learned selector. Moreover, Appendix M shows that it fails to outperform the best fixed policy, weakening the claim of adaptive structure selection.

  • While removing BMM reduces performance, comparisons with tuned fixed-threshold fusion methods are missing, leaving the benefits of probabilistic gating unclear.

  • Table 8 reports a latency of 3,990 ms per query for FluxMem on LoCoMo, higher than all reported baselines, raising concerns about practical efficiency.

Questions for authors

Can the authors compare BMM with tuned fixed-threshold fusion under identical settings?

Can the authors explain the higher end-to-end latency of FluxMem in Table 8?

Comments on the paper

16 comments
AuthorsOpenReview ↗

A gentle reminder for discussion

Dear Reviewer hNRr,

Thank you for your constructive comments and suggestions.

We have addressed your concerns by (1) clarifying FluxMem’s contribution in learning per-unit memory organization from downstream outcomes and using lightweight, distribution-aware BMM fusion; (2) adding end-to-end comparisons against random and feature-based selection, which FluxMem outperforms by 12.65 and 5.27 F1 points, respectively; (3) reporting a controlled threshold sweep, where BMM exceeds the best tested fixed threshold by 7.11 F1 points; and (4) explaining the efficiency results in Tables 7 and 10–12, including the 39% latency reduction with a 1.14-point F1 loss and the lowest reported latency in the other two evaluated settings.

The contribution clarifications and additional experimental comparisons have been incorporated into the revised manuscript, with changes marked in blue. Our efficiency response refers to measurements already included in the original submission.

We would appreciate your feedback on whether these explanations and results address your concerns. We would be happy to clarify any remaining questions.

Best regards,
The Authors of Submission 36391

AuthorsOpenReview ↗

A gentle reminder for discussion

Dear Reviewer oCvg,

Thank you for your constructive comments and suggestions.

We have addressed your concerns by (1) explaining how single-structure storage supports cross-structure retrieval while avoiding three indexes per unit, and clarifying that merged pages inherit the target session’s structure through incremental updates; (2) specifying the matching function and reporting a controlled threshold sweep, where BMM exceeds the best tested fixed threshold by 7.11 F1 points; and (3) directly addressing your identical-score example and explaining how BMM improves on a global cutoff using the current candidate distribution, supported by concrete examples and score-distribution statistics from ten LoCoMo conversations. These explanations and results have been incorporated into the revised manuscript, with changes marked in blue.

We would appreciate your feedback on whether our responses address your concerns. We would be happy to answer any further questions.

Best regards,
The Authors of Submission 36391

AuthorsOpenReview ↗

A gentle reminder for discussion

Dear Reviewer focy,

Thank you for your constructive comments and suggestions.

We have addressed your concerns by (1) providing end-to-end comparisons against homogeneous, random, heuristic, and oracle configurations, with FluxMem exceeding the heuristic by 5.27 F1 points and trailing the oracle reference by 2.34 points, alongside the selector accuracy reported in the frozen-pool pilot; (2) clarifying the backbone, evaluation protocol, and metric differences relevant to the published Mem0 and Zep results, including our use of the official Zep Cloud API; (3) pointing to Table 7, which reports 0.05 ms for selection with no additional LLM calls; and (4) clarifying that structure selection occurs at memory-writing time, with cross-dataset results and category-specific retrieval statistics showing how the stored structures support different query needs.

The expanded selection comparisons and the clarification of write-time selection versus query-time retrieval have been incorporated into the revised manuscript, with changes marked in blue.

We would appreciate your feedback on whether our responses address your concerns. We would be happy to answer any further questions.

Best regards,
The Authors of Submission 36391

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.