- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies online memory construction, where each new item must be ingested on arrival. Existing agentic memory systems call an LLM for every ingestion decision, so their cost grows with the corpus. Thus, the authors propose to grow a tree-structured memory through two learned local decisions per item, (1) routing the item from the root to a position in the tree, and (2) reorganizing a small window around it with IDLE, PROMOTE, DEMOTE or GROUP edits. Both decisions are made by a 0.5B policy distilled from a 27B teacher. Retrieval combines dense and lexical search, hand-designed tree factors and a reranker. On CloneMem, RealMem and NovelQA, the method outperformsHippoRAG 2 while using less than a tenth of the hosted-LLM tokens of the cheapest competitor.
Critical strengths
-
Turning online ingestion into small local decisions and distilling them into a 0.5B model is a simple and practical idea. It removes most per-item LLM calls, and the savings are large. The method uses over 10x fewer hosted-LLM tokens and builds its memory at least 8x faster than the agentic memory baselines. Table 6 also shows that the local window stays small as the tree grows, which supports the locality claim.
-
The distillation works well. On held-out decisions the 0.5B policy reaches 78.8% accuracy, close to the 27B teacher's 80.6%. Random legal actions (21.7%) and the untrained base model are far behind. This suggests a small model is enough for these local decisions.
-
The design is clean. Grammar-constrained decoding makes every edit legal by construction, so a bad output can't break the tree. The weighted loss focuses on the few tokens that actually carry the decision, which is a sensible fix for answers that are mostly template.
-
The paper is well written and easy to reproduce. Figures 1 and 2 explain the method clearly, and the appendix gives all the prompts, training settings and data sources.
Critical weaknesses
-
I am concerned that most of the gain may not come from the tree. By the paper's own numbers, BM25 and the reranker add about 7 to 16 points of R@10. The tree factors add less than 1 point (+0.92, +0.25, +0.18), and that gain is only significant on CloneMem. A flat retriever with the same BM25 and reranker would likely already beat all the baselines. So it is hard to credit the results to the structured memory, which is the main claim.
-
Meanwhile, the baselines don't use the same retrieval stack. The proposed method uses
Qwen3-Reranker-4B, but none of the memory baselines gets a comparable reranker. As a result, Table 1 compares retrieval pipelines as much as it compares memory organizations. -
The value of the learned policy isn't shown directly. The policy is only checked by its agreement with the teacher on 500 decisions. The paper never compares it with trees built in simpler ways, such as MemTree, RAPTOR or even random grouping, under the same retrieval. MemTree is described as the closest work but isn't used as a baseline.
-
The cost comparison is unfair and leaves out important parts. Teacher labeling took about 6,500 B200 GPU-hours and training used up to 40 B200 GPUs, but neither shows up in the main cost claims. The 4B reranker at query time isn't counted either, even though the abstract says retrieval needs no LLM calls.
-
The evaluation only uses R@10. There's no metrics for QA accuracy, etc. Mem0 and A-Mem are built to return compressed memories for answering questions, so judging them only by recall over mapped-back sources may not be fully fair to them, I think?
Questions for authors
I would be thankful and willing to raise my score if the authors can (partly) address the following questions/concerns.
-
Could add baselines like "Flat + BM25 + reranker" with no tree, and ideally give HippoRAG 2 the same reranker? If the tree still helps clearly on at least two benchmarks, my concern would be resolved.
-
Under the same retrieval, how does the learned tree compare with trees built by (1) the teacher, (2) by MemTree-style similarity insertion, or (3) by random grouping?
-
Most of the tree's gain is on latest-state questions and comes through F3. Would a simple recency boost without any tree give similar gains?
-
How well does the policy perform on the benchmark trees themselves, which differ from the training data? Does tree quality drop as more items arrive?
-
Could the authors report the full cost, including teacher labeling, training and the reranker? An estimate of how many ingested items it takes for the method to pay off would be better. Meanwhile, why is retrieval on RealMem so slow?
-
A small confusion I may have made. The gains in Sec. 4.2 are measured from Flat, but Table 2 uses Base (cosine), and the two start from slightly different numbers. Do they retrieve over different units or the same?
2 comments
Dear Authors,
Thank you for the comprehensive response and the extensive new experiments. I appreciate that you unified the retrieval stack across all baselines, provided the "Flat + BM25 + reranker" comparison, and clarified the training and labeling costs.
While these additions help address my primary concerns regarding the fair evaluation of the tree structure and the actual system overhead, after carefully considering the new data and the nuances of the results, I have decided to maintain my current score. However, given the additional complexities and technical details discussed in the rebuttal, I will be lowering my confidence score.
Regards,
Reviewer xcLQ
Response to Reviewer xcLQ (1/2)
We thank the reviewer for the careful review.
W1 & Q1: Is the gain due to the tree, rather than BM25 and the reranker?
Please see General Response A.
W2: The baselines do not use the same retrieval stack.
Now we are all fair. Please see General Response C.
W3 & Q2: How does the learned tree compare with simpler trees under the same retrieval: trees built by the teacher, by MemTree-style insertion, by random grouping, and by RAPTOR?
Table R1 in the General Response (B) compares the learned tree with these simpler trees under the same retrieval pipeline. Beyond the 500 judged decisions, we also measure agreement with the teacher on 18,104 held-out reorganization decisions (App. C).
Q3: Would a simple recency boost without any tree give similar gains, given that much of the gain appears on latest-state questions through F3?
Table R8. On latest-state questions (C3), R@10 of F3 on the learned tree, of an exponential recency boost (Park et al., 2023), and of F3 without the tree (it lifts the newest items of the whole stream); cosine base, no BM25 or reranker.
Q4: How well does the policy perform on the benchmark trees, which differ from the training data, and does tree quality drop as more items arrive?
The benchmark trees have no teacher labels, so we measure their quality by what the tree adds to retrieval. Table R9 splits the gain of the tree factors by the number of items N in the tree when each question was asked, on the two online benchmarks (NovelQA questions are asked only after a whole book has been inserted). It is a reasonable concern, in experiment we observed that the descent was not obvious.
Table R9. R@10 of Flat + BM25 + reranker (no tree) and of Ours, by the number of items N in the tree when the question was asked (averaged over the queries in each bucket). Ours: mean ± standard deviation over ten end-to-end runs.