acceptodds
Under review as a conference paper at ICLR 2027

Read the Order, Don't Extract It: Deriving Agent Memory Structure from the Bytes

Abstract

LLM agents depend on memory that is organized and served in real time. Their stores differ in what they already record: logs number their lines, example banks delimit their records, and prose does neither. Current systems either fix one configuration for every store or pay a language model to rebuild structure at write time. We introduce MUSTER, which reads that structure from the raw bytes instead. Two query-independent statistics, the share of lines that open with a serial marker and the share of text inside bounded records, set the retrieval unit, the parent hierarchy, whether an order may be used, and how retrieval slots are shared across channels. No generative model runs at write time, so a memory is ready in seconds as the store grows, and a query costs one ranking pass and one reader call. On MemoryAgentBench the same rule configures thirteen of fourteen stores in three seconds, and MUSTER scores 57.0, 60.1 and 64.0 at GPT-4o-mini, GPT-4.1-mini and GPT-5-mini. At the benchmark's GPT-4o-mini reader it is first among the retrieval and memory systems reported, 15.4 points above the strongest, and at GPT-4.1-mini it is 13.2 points above reading the whole store in context. On BEAM from 100K to 10M tokens, the rule's 512-token units score 18.6 points above unrefined 4096-token chunks at 100K, and MUSTER matches the strongest model-built memory under our main reader (74.9, 72.0 and 64.2 against 75.4, 72.3 and 62.4) while building each 10M-token memory in 36 seconds instead of 13 hours and 10,429 model calls, leads a hosted memory service by 7.8 to 15.3 points, and, read together with a model-built memory at that memory's write-time cost, scores above both. Agent memory can thus keep pace with the conversation and sit beneath or alongside model-built memories.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent memory, retrieval-augmented generation, long-context retrieval, memory systems, structure induction, hierarchical retrieval, AI agents, LLM agents, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HbWEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose a memory system for LLM agents. Instead of using an LLM to organize memory, MUSTER looks at the format of the stored text. It checks whether the text has numbered lines or clear record boundaries. Based on this information, it decides how to split and retrieve memory. The system also uses dense retrieval, BM25, and graph retrieval to find relevant information. Experiments show that MUSTER performs well on several benchmarks and takes much less time to build memory than LLM-based methods.

Critical strengths

The paper presents a simple and useful idea. Using the existing format of memory to organize information is interesting and can save a lot of time. The method does not need training or LLM calls to build memory. The experiments cover different tasks and show good results. I also like that the authors test the method on very large memory stores and compare its cost with other memory systems.

Critical weaknesses

The comparison with other methods may not be fully fair. Table 1 includes results from previous papers, and some methods use different reader models or only report results on certain tasks. MUSTER also uses multiple retrieval methods and a reranker. It is hard to tell whether the improvement comes from MUSTER's new idea or from these existing components.

The improvement from the proposed rules is not very large. Table 3 shows that using the proposed policy gate improves the overall score from 54.5 to 56.2. This is a relatively small improvement compared with the full system's performance. It is unclear how much the two text statistics contribute beyond a strong retrieval system.

The cost comparison is incomplete. The authors show that MUSTER builds memory much faster than other methods. However, the system still needs embeddings, indexing, and reranking. The paper does not clearly compare the total cost of running these systems, including the cost of answering questions.

Questions for authors

How does MUSTER work when the memory has no clear format or contains different types of text?

Can the authors compare MUSTER with other methods using the same retrieval pipeline and reader model?

How much of the improvement comes from the two text statistics rather than the other retrieval components?

2 comments
AuthorsOpenReview ↗

Response to Reviewer HbWE

Thank you for the review. Direct answers to Q3, Q2 and Q1. With the same retrieval components and reader, the derived structure and gate add 7.2 Overall points (Q3); with GPT-4o-mini for every system, MUSTER scores 57.5 against HippoRAG-v2's 43.7, the strongest released baseline in this comparison (Q2) [N]. The rule still sets units and parents for stores without serials (Q1) [S]. Does the control answer Q3? If a factor remains uncontrolled, please name it.

Q3 and W2: what the statistics add. Using the same retrieval, reranking, reader and assembly components, the derived structure and gate raise Overall from 50.3 to 57.5; AR, TTL, LRU and SF all improve, +7.2 Overall (Appendix N; paired 95% CI [4.9, 9.4], no task lower) [N]. The whole-chunk encoding and budget exceptions are stated in common part 3. It is reported separately from the submitted scores. Table 3's 54.5 to 56.2 isolates one decision of the rule; its caption reads "the serial gate against each fixed policy", with views held at prune [S]. The later control measures the other decisions too: 50.3 to 56.0 with units, parents and order under the balanced policy, then 57.5 with the gate. Every competency rises: AR 71.8 to 73.8, TTL 43.6 to 47.5, LRU 42.9 to 44.2, SF 43.0 to 64.5 [N].

The submission measures three components separately: the store-conditioned gate (+1.7), the record-preserving unit (+4.4 on MCC) and the read path over ranking alone (mean +3.6 over thirteen cells), in Section 1 of the submission and Tables 3, 17 and 16 [S]. Revised Appendix R preserves the original +4.4. Revised Section 5.1 reports +4.0 on MCC for the refreshed cut comparison [N]. Submitted Table 16 fixes candidates and cross-encoder in both arms.

Q2 and W1: same reader for every system, same pipeline for the control. Table 1's caption reads "at a GPT-4o-mini reader except where the row says otherwise, with the other systems as [1] reports them": the retrieval and memory rows use the benchmark's GPT-4o-mini reader except the explicitly labelled MIRIX row, the whole-store rows name their readers, and the FC-only row marks its limited scope [S]. After submission we also ran released systems ourselves on all fourteen cells with GPT-4o-mini and the same native scoring (table in common response, part 2): HippoRAG-v2 and True Memory, the strongest released systems in that table, score 43.7 and 43.6, the fixed-structure control 50.3, and MUSTER 57.5 [N].

The submission's MUSTER score is 57.0. The held-out comparisons in the common response use the same readers, judges and answer prompts for every system; implementations or hosted APIs are used as appropriate.

Q1: irregular stores. The rule still selects units and parents without serials. EventQA tests its prose branch [S]; BEAM (serial 0.18) tests messages at three scales [S]; LoCoMo tests unnumbered chat logs [N]. Numbered IDs need not mean supersession, which is why revised Section 3.5 explicitly scopes prune and retains all selected records under the label view.

W3: cost. Submitted Table 12 includes embeddings and BM25 and lists model load separately. Its read path uses one ranking pass and one reader call [S]. The revised numbered-fact walk can rerank up to 400 walk candidates across up to four rounds (Section 3.5) [N]. Revised Table 16 reports 77 s with 0 generative calls for a fixed 10M conversation index [N]. Appendix R preserves submitted timings and adds full reader inputs. Part 2 reports all 65 builds and Mem0's actual concurrency [N]. Matched-hardware per-query latency is unmeasured. Mem0's 10M build reads about 101M model-input tokens over 10,429 calls (submitted Table 4) [S].

We ask you to reconsider the rating in light of the matched-pipeline control.

AuthorsOpenReview ↗

Gently following-up

Dear Reviewer HbWE,

Thank you for raising the question of what MUSTER adds beyond the retriever and reranker. Our November 18 response points to revised Table 2: the fixed-structure control scores 50.3, selecting units, parents, and order from the stored text gives 56.0, and adding the store-specific channel policy gives 57.5 (+7.2 points; paired 95% CI [4.9, 9.4]). The retrieval components and reader are shared, with encoding-window and budget exceptions detailed in Appendix N.

Could you please indicate whether this comparison addresses your attribution concern, and which concerns, if any, remain decisive for your recommendation?

Thank you for your time and consideration.

Reviewer_L98vOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MUSTER, which derives memory organization from existing textual structure rather than generative extraction at ingestion. Two statistics—the prevalence of numbered lines and the token-weighted share of bounded records—select retrieval units, parent groups, order handling, and retrieval-channel allocations. A hybrid retriever and reranker construct a read-time view. For ordered records, slot-level consolidation favors the latest version without modifying the underlying store.

Critical strengths

Pros

  1. MUSTER reduces ingestion cost by avoiding generative extraction. It preserves source evidence while using embeddings and reranking for retrieval.

  2. Channel gating improves the overall score by 1.7 points over either fixed policy. With candidates fixed, direct slot reading approaches an entity-aware extractor without extra generation.

  3. The evaluation covers long histories, different readers, and complementary memory use.

Critical weaknesses

Cons

  1. Several design choices were tuned on the reported MemoryAgentBench questions.

  2. Surface order does not reliably show that a newer fact replaces an older one.

Questions for authors

Questions

  1. What evidence should justify replacing an older fact in the retrieval view?

  2. How should structure derivation adapt when a growing store changes format?

1 comment
AuthorsOpenReview ↗

Response to Reviewer L98v

Thank you for the review, and for recognising that MUSTER preserves source evidence while avoiding generative extraction. MUSTER uses both the serial gate and a shared entity-and-attribute slot key under the successive-value assumption stated in revised Section 3.5. Only the read-time view changes; the source memory is preserved. In the submission this applies to the two FactConsolidation stores and to no other of the fourteen (Figure 1a). BEAM runs with supersession off. With the same reader and prompt for every system MUSTER still leads Mem0 on contradiction resolution, 75.9 against 69.3 at 1M and 80.0 against 65.0 at 10M (Table 5, U = 20) [S]. Does this scope resolve W2, or is there a case you would like us to test?

W1: development and transfer. Appendix L separates the fixed statistics, thresholds and rule from choices developed on MemoryAgentBench, and the revised Section 4 lists them. The submitted BEAM evaluation uses the unchanged derivation rule [S]. The held-out LoCoMo and Refined runs reuse it [N].

On BEAM, MUSTER matches the strongest model-built memory under our main reader [S]; on LoCoMo and Refined it is first among memory systems in each reported column [N].

W2 and Q1: when to replace an older value. As your second strength notes, direct slot reading approaches an entity-aware extractor. On fixed FactConsolidation selections, without refill and with a temperature-zero reader, byte-read slots score 41/92 on multi-/single-hop, against the entity-aware GPT-4.1-mini extractor's 43/93 with one generative call per question [S]. Shuffling serials gives 33/59, below showing no order at 35/64. This measures what recorded order contributes under the benchmark's successive-value assumption.

At read time the serial gate must apply and the records must share the entity-and-attribute slot key. The source is never rewritten. For numbered data outside the successive-value assumption, the label view retains every selected record. Submitted Appendix F reports its trade-off on the two FactConsolidation cells: 0.400 versus prune's 0.380 on multi-hop, and 0.770 versus 0.910 on single-hop.

The revised Section 3.5 states: "Prune assumes successive values of single-valued slots. For numbered stores outside that assumption, the label view keeps every selected unit and marks the newest. The view is selected explicitly." In submitted Appendix F the label view tags the newest unit [CURRENT] and the older ones [SUPERSEDED].

Q2: growing stores. Both statistics can be recomputed from the current text as the store grows. A format change can then select different settings without a generative call. Our measurements use fixed stores. On BEAM stores of three sizes, from 100K to 10M tokens, the rule is read once per scale and keeps the same reading (pooled recshare 0.986 at 100K, 0.993 at 1M and 10M, Appendix C) [S]. The measured empty threshold intervals, 0.998 for serial and 0.425 for recshare, describe MemoryAgentBench's observed stores [S].

Complementary memory use. MUSTER can "sit beneath or alongside model-built memories" (Abstract): its units followed by Mem0's memories score 78.6, 74.1 and 65.2 on BEAM, "above either system at the same budget" (Section 5.3) [S]. Revised Table 6 applies the same rule to both stores. Read together at 19.5K tokens, they average 73.1 across the three scales, against 71.3 for the conversation and 72.0 for Mem0's memories alone. With relevant original turns in the same context, the reader can check a note against what was said [N].

We ask you to reassess W1 and W2 in light of the held-out transfer and the stated scope of read-time replacement.

Reviewer_ZELjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MUSTER, a method that uses two query-independent byte statistics, serial and recshare, to determine an agent memory’s retrieval unit, parent hierarchy, use of recorded order for read-time conflict resolution, and channel policy, aiming to avoid both a single fixed configuration for all memory stores and costly LLM-based restructuring at write time. The construction of memory requires no generative calls at write time. At read time, it employs a hybrid retriever whose candidates are reranked by a single cross-encoder, and each query is processed with only one reader call. MUSTER automatically configures most of MemoryAgentBench stores and outperforms the retrieval and memory systems reported by the benchmark. On the BEAM benchmark, MUSTER achieves accuracy comparable to Mem0 without re-tuning, while with a significantly lower construction cost.

Critical strengths
  1. The underlying rule is straightforward: before spending time or resources on building a new memory structure, the method first checks whether the existing store is already organized well enough to be used directly.

  2. The method requires no generative LLM calls at write time, so constructing the memory is much less expensive than with systems such as Mem0 that rely on LLMs.

  3. Conflicts are resolved at read time, rather than by rewriting the memory store itself. As a result, an incorrect resolution never causes evidence to be lost permanently.

Critical weaknesses
  1. The method only works as intended if the memory format follows the neat assumption, such as numbered lines or blank sperated records. The thresholds were only validated on the 14 stores of one benchmark and could perform totally differently in other cases. If the data is not organized in such format, the main mechanism no longer works and falls back to RAG. Some counterexamples could easily trigger the system in the wrong way, e.g., a store consisting of numbered steps or numbered IDs, where the numbers do not indicate temporal order.

  2. Several design choices were tuned on eval benchmark questions. The improvements mainly come from the numbered format of one synthetic dataset and from an extra reranking step that the baseline systems don’t have. Every result is just a single run, so there is no information about how consistent the system’s performance is.

  3. The released version actually performs worse than the basic setup for two out of three readers. The claimed advantage of better performance with bigger memory stores doesn’t hold up under matched retrieval budgets, and scoring above both MUSTER and Mem0 requires combining with Mem0 and paying its full construction cost.

  4. The reported time of 36s only includes a single conversation and does not count the time it takes to load the models. The concurrency of Mem0 is not reported. Furthermore, the paper does not mention how long it takes to answer queries, as MUSTER moves some of the work to read time, the end-to-end time comparison is necessary.

  5. For the BEAM benchmark, the authors use a special prompt that gives order information, even though the rule itself decides that no usable order exist. When using the official prompt, MUSTER only matches Mem0’s performance. Also, comparing to 4096-token chunks isn’t straightforward because the encoder cuts off most of each chunk.

  6. The paper says its constants weren’t hand-picked, so Section 5 is meant to show how the rule works, not to fine-tune its settings. But in Appendix I, the authors admit that a few other parts, like floor triples, view, renderer, and K1, were actually chosen based on the evaluation questions.

Questions for authors

Most of my questions and concerns are included in the weakness section. Specifically, I would like to ask the authors:

  1. How does MUSTER perform when handling with irregular or mixed-format memories?

  2. Can you share the mean and std across multiple runs with different random seeds?

  3. Could you provide the end-to-end per-query latency and token cost, as well as the total construction time for all conversations, using the same concurrency setup?

  4. How big is the difference between using 4096-token chunks and using fine-grained units, especially if a long context encoder that doesn’t truncate is used?

  5. Where does the serial information in prompt C come from in BEAM main setting?

  6. The two FC cells show identical scores across base, +extraction, and shipped configurations for all three readers (at t =0.7). Were these cells re-run, or were the scores reused from earlier runs?

  7. Do the conclusions hold when evaluated on held-out questions or datasets that were not used for any design choice?

1 comment
AuthorsOpenReview ↗

Response to Reviewer ZELj

Thank you for the review and for recognising the check-first, generative-free construction. With the same retrieval components and reader, MUSTER scores 57.5 against the fixed-structure control's 50.3 (95% CI of the gain [4.9, 9.4]); common part 3 states encoding-window and budget exceptions [N]. The submitted rule transfers unchanged to 65 BEAM conversations [S]. Which specific soundness concern remains after this control and the scoped prune revision?

W1, Q1 and Q7: formats and transfer. BEAM is a second submitted benchmark: 65 conversations at three scales, with no rule choice revisited [S]. On held-out LoCoMo and Refined, MUSTER is first among memory systems in every reported column [N]. Unnumbered stores still get units and parents from the rule (LME aside) [S].

In virtual BEAM-100K assistant-list stores, 491 of 21,308 lines (2.3%) differ from the highest-order entry chosen for their slot in the diagnostic [N]. The revised Section 3.5 scopes numbered stores: prune assumes successive values of single-valued slots; outside that assumption, the explicitly selected label view keeps all selected units. No view rewrites the store. Adopting the label view on both FactConsolidation cells costs 1.6 Overall against a 15.4-point margin (Appendices F and I) [S].

W2 and Q2: attribution and repeats. The submission gives the attribution itself: the read path adds a mean +3.6 points over ranking the same candidates by the cross-encoder alone (Table 16), and "the rest of the margin over the strongest reported system comes from retrieving over the stored text itself with a hybrid retriever and one cross-encoder instead of over structure a model built" (Section 5.1) [S]. Our contribution is the rule that jointly sets units, parents, usable order and policy from stored structure. Outside the SF competency the lead over HippoRAG-v2 is 6.5 of the 15.4 points [S]. Repeats: InfBench-Sum F1 0.4993/0.4764 on 91 identical prompts [S]; DetectiveQA 67.6/66.2/64.8% on the same 71 contexts [N].

W3: Base and shipped. Submitted Table 2's caption states that the ablation rows "were measured with one additional generative call per query, which the shipped system omits". Shipped scores 57.0/60.1/64.0 without that call, against Base's 56.2/60.4/64.1 with it [S]. With extraction held fixed, refinement adds 0.3/0.6/0.5 Overall. MUSTER alone matches Mem0 with no write-time model; for a system that already runs Mem0, adding MUSTER's units gives 78.6/74.1/65.2, "above either system at the same budget", with no generative calls to build our source index (Section 5.3) [S]. The revision removes the Conclusion's growth sentence.

W4 and Q3: cost. Submitted Table 12 reports a 36 s 10M build plus 11.5 s load. Mem0 writes its memories in 13 h and 10,429 calls (Table 4) [S]. Revised Table 16 times a fixed conversation index at 77 s and 0 generative calls with a 2048-token encoder cap [N]. Part 2 gives total build time, concurrency and reader inputs [N]. Matched-hardware per-query latency is unmeasured.

W5 and Q5: prompt and source order. Prompt C was written for MUSTER's contexts on BEAM's separate 500K development conversations. It asks readers to use recorded order and state dates, and is applied unchanged to every system. The order marks in these contexts are BEAM's own message IDs, recorded session/turn fields where available, and time anchors on under 2% of BEAM's source messages, not inferred labels [N]. Low serial disables our supersession; it does not remove BEAM's recorded fields. Mem0 and Honcho contexts also carry order: numbered dated memories and timestamped observations, in 400 of 400 at 100K [N].

The review states that under the official prompt "MUSTER only matches Mem0's performance": at U = 40 (19.5K tokens; Mem0: 8.2K) it scores 60.5/57.1/49.6 against Mem0's 60.4/58.0/49.6, with no write-time generation (Tables 4 and 13) [S]. With Qwen3.5-9B under prompt A, MUSTER at U = 20 has the highest mean of seven systems: 44.2 versus Mem0's 41.2 at the same 9.7K [N]. Paired intervals exclude zero against Mem0, Honcho, Hindsight and EverMemOS [N].

W5 and Q4: chunk truncation. The archived whole-chunk configuration sets encoder and reranker windows to 8192; two and four chunks score 62.6/70.1 instead of 56.3/61.3 [N]. The submitted fine-unit conditions remain ahead by 12.3 [8.3, 16.0] and 5.7 [2.6, 8.9] (Appendix N) [N].

W6: constants. Appendix L separates constants fixed before reported runs from read-path choices developed on MemoryAgentBench; the revised Section 4 lists both.

Q6: identical FC cells. Compensated refinement does not apply to FC's numbered records (7 to 30 tokens, Appendix A), and its structural path never reads the extraction view. Inputs are identical across the three columns. Each reader's FC cells were read once and reused, not counted as independent rereads; Appendix B now says so.

We ask you to reconsider the rating against this evidence.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common response (1/3)

We thank the reviewers. MUSTER automatically configures retrieval for heterogeneous stores. Two query-independent byte statistics replace manual setup of units, parents, usable recorded order and channel policy without a generative call [S]. At the same 19.5K-token budget, RAG's source and existing notes score higher together on average (revised Table 6) [N]. With the same retrieval components and reader, its rule raises Overall from 50.3 to 57.5 (+7.2, 95% CI [4.9, 9.4]); part 3 states encoding-window and budget exceptions [N]. The rule transfers unchanged to BEAM [S]. Part 3 checks the submitted evidence on transfer and repeated runs. [S] uses the submitted PDF's evidence, numbering and quoted text; [N] marks follow-ups.

What is new: detect before you construct [S]. An agent inherits stores it did not design: "a transcript, a list of assertions, a novel, a bank of labelled examples" (Section 1). "A fixed chunk size treats every store alike." Model-built memories pay a model at write time, and EdgeMem "builds memory without a model in one fixed configuration". "MUSTER adds the missing decision: it reads from the bytes whether the store already records the structure a reader needs, and then sets the unit, the hierarchy, the use of order and the channel policy from it" (Section 1). The nearest methods, Adaptive Chunking, Mix-of-Granularity and HiChunk, select chunkers: "All three choose among chunkers, while ours decides whether the store exposes an order to read" (Section 2). Our original contribution is the rule that jointly makes these structural decisions. The reviews recognise this idea: "the method first checks whether the existing store is already organized well enough to be used directly" (ZELj) and "Using the existing format of memory to organize information is interesting" (HbWE).

Automatic configuration across heterogeneous stores [S]. The two statistics "see no question, gold answer or dataset name and call no model; over fourteen stores they run in three seconds and configure thirteen" (Section 1), across "a range of 733× that one threshold pair has to cross without per-dataset tuning" (Section 4). On eight paired cells, "the rule's call is better on six, within 0.010 on one, and misses one" (submitted Figure 1, Table 17) [S]. Revised Figure 2 gives five better, two within 0.010 and one worse [N].

Neither fixed channel policy serves both kinds of store. The submission measures it: each fixed channel policy scores 54.5 and the rule's gate 56.2 (Table 3), because "both constants land at 54.5 while trading 5.1 AR points for 7.0 SF points" (Appendix G). Reviewer L98v records the same result: "Channel gating improves the overall score by 1.7 points over either fixed policy" [S]. The follow-up control repeats the pattern: fixed structure 50.3, the statistics with either policy applied everywhere 56.0 or 55.5, MUSTER 57.5, with every competency rising (part 2) [N].

The decisions carry the results.

  • Order read from the bytes [S]: on fixed selections, byte-read slots score 41 and 92 with no model call, an entity-aware extractor 43 and 93 with one call per question; shuffling the same serials gives 33 and 59, below showing no order at all (35 and 64) (Section 5.1, Appendix E).
  • Units chosen by the rule on a second benchmark: on BEAM-100K its 512-token record units beat unrefined 4096-token chunks by 18.6 points (95% CI [15.1, 21.9]) [S], and by 12.3 [8.3, 16.0] when the chunks are encoded whole [N].

What it delivers.

  • First among the retrieval and memory systems reported by MemoryAgentBench on all four competencies, 57.0 Overall against 41.6 (Table 1), and above the whole store read in context at the same reader, by 13.2 at GPT-4.1-mini (1M tokens) and 3.5 at GPT-5-mini (400K) (Section 5.1) [S].
  • On BEAM, 65 conversations from 100K to 10M tokens, the unchanged rule matches Mem0, the strongest model-built memory (pooled +0.4, 95% CI [-1.3, +2.0]), and builds a 10M-token memory in 36 seconds against 13 hours and 10,429 generative calls (Section 5.3) [S]. On held-out LoCoMo and LoCoMo-Refined it is first among memory systems in all four score columns (revised Table 3) [N].
  • With GLM-5.3-Flash, MUSTER scores 70.03 on MemoryAgentBench; the additional result comment gives all four competencies, the True Memory comparison and the numbered-store read added after submission [N].

It composes with model-built memory [S]. "MUSTER's U=20 units followed by Mem0's memories, 19.5K tokens in all, score 78.6, 74.1 and 65.2, above either system at the same budget (+3.1, +0.8, +1.3), at Mem0's write-time cost" (Section 5.3). Agent memory can "sit beneath or alongside model-built memories" (Abstract).

Evidence stays grounded [S]. Retrieved evidence is the stored text itself, in stored order per container. Slot and order cues come from recorded text. Conflict resolution changes only the read-time view; the source memory is never rewritten.

AuthorsOpenReview ↗

Common response (2/3): results

Results (common response, part 2 of 3) [N]. With the same retrieval components and reader, the rule raises Overall from 50.3 to 57.5 (+7.2, 95% CI [4.9, 9.4]). MUSTER is first among memory systems in all four held-out chat columns. The 57.5 is the GPT-4o-mini attribution result, distinct from the submitted 57.0 and the GLM-5.3-Flash result (70.03); part 3 states the settings and frozen renderings.

MemoryAgentBench, one reader for every system [N]. After submission we ran released systems ourselves on all fourteen cells with GPT-4o-mini and the benchmark's native scoring:

SystemOverall
Mem0 2.1.026.7
Honcho35.0
Hindsight 0.10.331.4
EverMemOS, GPT-OSS-120B build23.1
Mem0 on raw chunks38.9
HippoRAG-v2, GPT-OSS-120B build43.7
True Memory43.6
Fixed-structure control, same retrieval components50.3
MUSTER (ours)57.5

In this comparison contexts retain at least one whole unit; on FactConsolidation each system reads its own uncut retrieval.

The last two rows share retrieval, reranking, reader and assembly components; part 3 states the whole-chunk encoding and budget exceptions. The 50.3 row uses a fixed retrieval configuration; the rule raises it to 57.5, and every competency rises: AR 71.8 to 73.8, TTL 43.6 to 47.5, LRU 42.9 to 44.2, SF 43.0 to 64.5 (revised Table 2, Appendix N). With either channel policy applied everywhere the statistics give 56.0 or 55.5; the rule's gate gives 57.5. Over the same 1,671 questions the gain over the control is 7.2 points (95% CI [4.9, 9.4]; no task lower), and MUSTER leads HippoRAG-v2 by 13.8 (95% CI [11.2, 16.4]; higher on all fourteen tasks) and True Memory by 13.9 (95% CI [11.3, 16.5]); no resample of 10,000 falls at or below zero [N]. With the benchmark's original Overall weights, the twelve tasks outside FactConsolidation contribute 1.8 points of the control gap (95% CI [0.3, 3.4]); their contributions to the HippoRAG-v2 and True Memory gaps are 5.4 [3.6, 7.2] and 4.3 [2.5, 6.1] [N].

Held-out chat logs (revised Table 3) [N]. LoCoMo has ten conversations and 1,540 questions; LoCoMo-Refined corrects the same conversations' answers (1,382 questions). No method choice was revisited. With serial 0 and recshare 1.000 the rule takes 512-token record units on its own, read at MemoryAgentBench's budget of ten units.

SystemLoCoMo, Mem0's promptLoCoMo, EverMemOS's promptRefined, GPT-4o-miniRefined, GLM-5.3-FlashGenerative calls to build and search (LoCoMo)
Mem0 2.1.080.5089.4473.7380.545,882
Honcho67.3477.6256.4465.56not observable
Hindsight 0.10.373.4885.3563.6870.91602
EverMemOS83.8793.0575.3380.329,398
HippoRAG-v275.0984.4667.5876.052,073
True Memory, with HyDE81.9390.5871.5680.101,540
Whole conversation, 31.1K tokens82.0693.4472.0788.930
MUSTER, about 5K tokens86.0493.4677.5788.780

Under Mem0's own prompt MUSTER leads the other memory systems by 2.2 to 18.7 points; with GLM on Refined, by at least 8.2. On LoCoMo it reads 5.0K tokens where the whole conversation is 31.1K, with no generative call to build or search. GPT-4.1-mini reads every LoCoMo context and GPT-4o-mini judges; Refined is scored with the official Qwen3-14B. We used released implementations or hosted APIs. For Refined, model-built stores except Honcho's were rebuilt with GPT-OSS-120B.

BEAM [S]. 65 conversations (20, 35 and 10; Appendix C) at 100K, 1M and 10M tokens, with no rule choice revisited: MUSTER matches Mem0 (pooled +0.4, 95% CI [-1.3, +2.0]) and leads Honcho by 7.8 to 15.3 points (Section 5.3, Table 4).

Both stores [N]. At 19.5K tokens, reading the conversation and Mem0's memories together averages 73.1 across BEAM's three scales, against 71.3 and 72.0 alone (revised Table 6).

Build and query cost. Table 12 includes statistics, record units, embeddings and BM25; its footnote lists the separate 11.5 s model load [S]. The submitted path uses one ranking pass and one reader call (Abstract) [S].

  • All 65 BEAM conversations (149M source tokens) build in 1.9 hours on Apple M3 Ultra fp16, one at a time after a one-time 2.2 s encoder load [N]. The submitted RTX PRO 6000 timings are retained in revised Appendix R.
  • Revised Table 16 reports 77 s and 0 generative calls for a fixed BEAM-10M conversation index with a 2048-token encoder cap, excluding encoder load and warm-up [N]. Mem0's submitted write took 13 h and 10,429 calls [S]. This was conversation 0 of ten that ran concurrently and finished in about 14.2 h [N].
  • Mean full reader input per query (GLM-5.3-Flash, prompt C) [N]:
System100K1M10M
MUSTER U = 2010.3K10.3K10.4K
Mem09.3K8.8K9.1K
Honcho15.5K21.2K15.3K
AuthorsOpenReview ↗

Common response (3/3): submitted evidence and settings

Submitted evidence on format transfer and repeated measurements. The two runs reported in Section 4 and Appendix H directly contradict the claim that every result came from a single run. The table also shows how the rule handles prose and where it transfers.

ZELj's statementEvidence and clarification
"The thresholds were only validated on the 14 stores of one benchmark and could perform totally differently in other cases" (W1)The unchanged rule was run on a second benchmark of 65 conversations: "On BEAM from 100K to 10M tokens it transfers without re-tuning" (Abstract). The review's own summary says MUSTER "achieves accuracy comparable to Mem0 without re-tuning" there. This tests unchanged-rule transfer; the threshold-separation analysis itself uses the fourteen MemoryAgentBench stores.
"If the data is not organized in such format, the main mechanism no longer works and falls back to RAG" (W1)The rule still decides there: "A store whose unit stays coarse is refined fourfold at read time" (Section 3.2), and "EventQA begins from base scores of 74.0, 71.0 and 76.0 and reaches 80.0, 80.0 and 82.0" (Section 5.1).
"Every result is just a single run, so there is no information about how consistent the system's performance is" (W2)"Two runs of one configuration give 91 of 100 questions a byte-identical context, none an identical summary, and judged F1 differing by 0.023" (Section 4, Appendix H). Table 4's caption reads "Mem0: mean of two searches", and BEAM parity carries an interval (pooled +0.4, 95% CI [-1.3, +2.0], Section 5.3).

If a soundness concern remains after this evidence, please name the specific check. We ask for reassessment against these results.

What was fixed in advance [S]. The concern about development on the benchmark rests on our own Appendix I. It separates pre-run statistics, thresholds 0.80 and 0.90, the rule and fourfold refinement from MemoryAgentBench choices (floor triples, prune, renderer, K1 = 200 and removal of extraction). Alternatives change Overall by 1.6 for the view, at most 0.71 for the renderer and at most 0.9 for extraction, versus a 15.4-point Table 1 margin. The rule transfers unchanged to BEAM [S] and held-out chats [N]. Revised Appendix L records BEAM container and two-store assembly comparisons. Prompt C was written on separate 500K BEAM conversations absent from reported scales.

Measured repeats. Submitted Appendix H repeats one InfBench-Sum configuration with GPT-5-mini and the benchmark's GPT-4o judge. On 91 of 100 byte-identical prompts, judged F1 is 0.4993 and 0.4764 (per-question SD 0.152, p = 0.165) [S]; the two runs average 0.488 (SD 0.016) [N]. Three GPT-4o-mini reads of the same 71 DetectiveQA contexts score 67.6, 66.2 and 64.8% (mean 66.20%, sample SD 1.41 points) [N]. Two GPT-4o-mini Refined reads on identical 1,382 prompts score 77.57/77.13% (mean 77.35%, sample SD 0.31 points) [N]. BEAM-100K has two 400-question GLM reads at 75.35/74.62% (macro mean 74.98%, sample SD 0.52 points) [N]. These are cell-level repeats. Submitted-length reconstruction also matches saved contexts byte for byte on SH-QA (100/100) and BEAM-100K (40/40 checked contexts) [N].

Configuration checks [S]. Base / +extraction / shipped: GPT-4o-mini 56.25/56.57/57.00; GPT-4.1-mini 60.35/60.99/60.09; GPT-5-mini 64.12/64.61/63.97 (Tables 8 to 10). The ablations use one additional generative call. These test configuration sensitivity.

Follow-up settings [N]. The GPT-4o-mini attribution and chat tables use frozen evaluation renders with 2048-token encoder/reranker windows and one block per parent for coarse stores; saved FC contexts contain no [CURRENT] tags. The fixed-structure control encodes its coarse chunks whole with an 8192-token window; its per-question budget rule retains at least one whole unit, with ten uncut 512-token units on FC. Its LME results reuse identical inputs. The Qwen3.5-9B prompt-A comparison uses the stated 9.7K follow-up condition. The 8192-token whole-chunk experiment is a separate control. Numbered-list and order-marker audits inspect the submitted source snapshots. The added GLM result is a separate full-population evaluation. The method name is MUSTER; these labels identify experiments.

Revised PDF. Sections 3.2 and 4 distinguish fixed constants from choices developed on MemoryAgentBench; Section 3.5 states prune's scope and the explicit label option. The Conclusion drops the growth sentence and describes fixed-store setup. Appendix B explains FC answer reuse; Appendix C states the serialized BEAM format and retains time anchors where available. Appendix R preserves the submitted tables and their model-load disclosure, with full reader inputs. The 18.6-point coarse-unit result is identified with its submitted encoder window. Appendix N reports the whole-chunk and matched-pipeline controls; Appendix O gives additional closed results. Cohorts are identified separately.

AuthorsOpenReview ↗

Additional result: full-population GLM evaluation

Full-population MemoryAgentBench result with GLM-5.3-Flash [N]. MUSTER scores 70.03 Overall, 11.56 points above True Memory under the same declared reader settings and the same GPT-4o judging (95% interval [9.2, 13.9]). This is distinct from the submitted GPT-4o-mini result (57.0) and the GPT-4o-mini attribution experiment (50.3 to 57.5).

The full fourteen-task population contains 1,671 questions and 1,813 reader evaluations, with three reads on DetectiveQA. The benchmark's native aggregation gives:

CompetencyMUSTER (%)
Accurate Retrieval85.58
Test-Time Learning54.24
Long-Range Understanding63.31
Selective Forgetting77.00
Overall: mean of the four competencies70.03
GLM-5.3-Flash evaluationOverall (%)
MUSTER70.03
True Memory, same GPT-4o judging58.48

The True Memory comparison uses the same 1,671 question identities and the same declared GLM reader settings: Morph FP8, high reasoning, temperature 0 and seed 42. The interval uses stratified paired bootstrap resampling of questions within each task. MUSTER is higher on thirteen tasks and equal on one. Independent recounts reproduce both Overall values and the point difference, which is calculated before rounding.

For LME and summarization we rejudged True Memory's saved answers with the GPT-4o-2024-08-06 protocol used for MUSTER, keeping every reader answer unchanged (400 questions, 600 judging steps). The interval compares saved outcomes question by question, not independent reader seeds. The twelve tasks scored without a judge account for 9.18 of the 11.56 points.

The 70.03 GLM evaluation uses the numbered-fact walk developed on FactConsolidation after submission. In the GLM version comparison, selective forgetting rises from 69.0 to 77.0, and the other three competencies move by less than one point each. The archived collection evaluations with this read score 60.6, 62.2 and 67.0 with the three readers of submitted Table 2 (57.0, 60.1 and 64.0). For those readers and Luna, revised Table 1 refreshes five ICL cells and reuses the other archived cells, including FactConsolidation (Appendix B). For GPT-4o-mini, that refresh gives 60.7 from 60.6 while retaining the two numbered-cell scores, 93.0 and 60.0. The four ICL-refresh rows are 60.7, 62.6, 67.3 and 67.08. The numbered-fact walk applies to qualifying numbered stores. BEAM and the held-out chats use their evaluated read paths without this numbered-store addition. Appendix R preserves the submission, and Appendix O reports the later matched-judging GLM pair (70.03 and 58.48). Its reported competitor aggregates follow the official benchmark table, including 0.1-point display corrections. Revised Table 7 uses 70.03 for MUSTER with GLM and retains the archived 67.08 with Luna. MUSTER's six-score mean is 72.26. True Memory's GLM 58.27 in Table 7 uses archived judging. Table 1 and Appendix O give the matched 58.48. The paired tests retain the archived cohort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.