- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper propose a memory system for LLM agents. Instead of using an LLM to organize memory, MUSTER looks at the format of the stored text. It checks whether the text has numbered lines or clear record boundaries. Based on this information, it decides how to split and retrieve memory. The system also uses dense retrieval, BM25, and graph retrieval to find relevant information. Experiments show that MUSTER performs well on several benchmarks and takes much less time to build memory than LLM-based methods.
Critical strengths
The paper presents a simple and useful idea. Using the existing format of memory to organize information is interesting and can save a lot of time. The method does not need training or LLM calls to build memory. The experiments cover different tasks and show good results. I also like that the authors test the method on very large memory stores and compare its cost with other memory systems.
Critical weaknesses
The comparison with other methods may not be fully fair. Table 1 includes results from previous papers, and some methods use different reader models or only report results on certain tasks. MUSTER also uses multiple retrieval methods and a reranker. It is hard to tell whether the improvement comes from MUSTER's new idea or from these existing components.
The improvement from the proposed rules is not very large. Table 3 shows that using the proposed policy gate improves the overall score from 54.5 to 56.2. This is a relatively small improvement compared with the full system's performance. It is unclear how much the two text statistics contribute beyond a strong retrieval system.
The cost comparison is incomplete. The authors show that MUSTER builds memory much faster than other methods. However, the system still needs embeddings, indexing, and reranking. The paper does not clearly compare the total cost of running these systems, including the cost of answering questions.
Questions for authors
How does MUSTER work when the memory has no clear format or contains different types of text?
Can the authors compare MUSTER with other methods using the same retrieval pipeline and reader model?
How much of the improvement comes from the two text statistics rather than the other retrieval components?
2 comments
Gently following-up
Dear Reviewer HbWE,
Thank you for raising the question of what MUSTER adds beyond the retriever and reranker. Our November 18 response points to revised Table 2: the fixed-structure control scores 50.3, selecting units, parents, and order from the stored text gives 56.0, and adding the store-specific channel policy gives 57.5 (+7.2 points; paired 95% CI [4.9, 9.4]). The retrieval components and reader are shared, with encoding-window and budget exceptions detailed in Appendix N.
Could you please indicate whether this comparison addresses your attribution concern, and which concerns, if any, remain decisive for your recommendation?
Thank you for your time and consideration.
Response to Reviewer HbWE
Thank you for the review. Direct answers to Q3, Q2 and Q1. With the same retrieval components and reader, the derived structure and gate add 7.2 Overall points (Q3); with GPT-4o-mini for every system, MUSTER scores 57.5 against HippoRAG-v2's 43.7, the strongest released baseline in this comparison (Q2) [N]. The rule still sets units and parents for stores without serials (Q1) [S]. Does the control answer Q3? If a factor remains uncontrolled, please name it.
Q3 and W2: what the statistics add. Using the same retrieval, reranking, reader and assembly components, the derived structure and gate raise Overall from 50.3 to 57.5; AR, TTL, LRU and SF all improve, +7.2 Overall (Appendix N; paired 95% CI [4.9, 9.4], no task lower) [N]. The whole-chunk encoding and budget exceptions are stated in common part 3. It is reported separately from the submitted scores. Table 3's 54.5 to 56.2 isolates one decision of the rule; its caption reads "the serial gate against each fixed policy", with views held at prune [S]. The later control measures the other decisions too: 50.3 to 56.0 with units, parents and order under the balanced policy, then 57.5 with the gate. Every competency rises: AR 71.8 to 73.8, TTL 43.6 to 47.5, LRU 42.9 to 44.2, SF 43.0 to 64.5 [N].
The submission measures three components separately: the store-conditioned gate (+1.7), the record-preserving unit (+4.4 on MCC) and the read path over ranking alone (mean +3.6 over thirteen cells), in Section 1 of the submission and Tables 3, 17 and 16 [S]. Revised Appendix R preserves the original +4.4. Revised Section 5.1 reports +4.0 on MCC for the refreshed cut comparison [N]. Submitted Table 16 fixes candidates and cross-encoder in both arms.
Q2 and W1: same reader for every system, same pipeline for the control. Table 1's caption reads "at a GPT-4o-mini reader except where the row says otherwise, with the other systems as [1] reports them": the retrieval and memory rows use the benchmark's GPT-4o-mini reader except the explicitly labelled MIRIX row, the whole-store rows name their readers, and the FC-only row marks its limited scope [S]. After submission we also ran released systems ourselves on all fourteen cells with GPT-4o-mini and the same native scoring (table in common response, part 2): HippoRAG-v2 and True Memory, the strongest released systems in that table, score 43.7 and 43.6, the fixed-structure control 50.3, and MUSTER 57.5 [N].
The submission's MUSTER score is 57.0. The held-out comparisons in the common response use the same readers, judges and answer prompts for every system; implementations or hosted APIs are used as appropriate.
Q1: irregular stores. The rule still selects units and parents without serials. EventQA tests its prose branch [S]; BEAM (serial 0.18) tests messages at three scales [S]; LoCoMo tests unnumbered chat logs [N]. Numbered IDs need not mean supersession, which is why revised Section 3.5 explicitly scopes prune and retains all selected records under the label view.
W3: cost. Submitted Table 12 includes embeddings and BM25 and lists model load separately. Its read path uses one ranking pass and one reader call [S]. The revised numbered-fact walk can rerank up to 400 walk candidates across up to four rounds (Section 3.5) [N]. Revised Table 16 reports 77 s with 0 generative calls for a fixed 10M conversation index [N]. Appendix R preserves submitted timings and adds full reader inputs. Part 2 reports all 65 builds and Mem0's actual concurrency [N]. Matched-hardware per-query latency is unmeasured. Mem0's 10M build reads about 101M model-input tokens over 10,429 calls (submitted Table 4) [S].
We ask you to reconsider the rating in light of the matched-pipeline control.