- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MemoryLOD, a memory framework for long-horizon language agents. It organizes source records into a graph with multiple levels of detail. The navigation scene presents detailed previews near the current focus and coarse summaries of other regions. Each visit automatically changes the allocation of detail. Routing digests are constructed independently of the question and require no language-model calls. Source openings provide the evidence for answering. A bounded review stage can request additional evidence and revise the answer. The evaluation covers BEAM, LongMemEval-V2, and AMA-Bench. Controlled comparisons support the complete scene policy. Gains over Hybrid RAG are concentrated in LongMemEval-V2. The paper also reports source replay and diagnostic ablations.
Critical strengths
1.The paper proposes a multiresolution navigation framework for agent memory. It improves answer quality over Local Tree across conversational, multimodal, and agent-history tasks. The gains are especially large on LongMemEval-V2. The controlled comparison uses the same model and underlying memory graph. These results identify memory presentation as an important factor in answer quality. This finding provides a useful design principle for agent memory interfaces.
2.The paper evaluates the review stage across several memory settings. Review improves accuracy on both LongMemEval-V2 settings. It reduces AMA-Bench accuracy from 70.00% to 64.40%. These results reveal substantial task variation in the effect of review. The analysis provides an empirical basis for assessing the benefits and limits of review in memory agents.
Critical weaknesses
1.The published comparisons in Figures 4–6 use different models and evaluation settings. The matched comparisons include Local Tree and Hybrid RAG. Local Tree is an internal scene variant. Hybrid RAG uses one reader call. The experiments do not include matched results for established navigation methods such as MemWalker or NaviRAG. The gains over Hybrid RAG on BEAM 100K and AMA-Bench also have paired 95% intervals that include zero. These results provide limited evidence for the method’s advantage over competitive navigation approaches under comparable inference budgets.
2.On AMA-Bench, accuracy decreases from 70.00% in the first draft to 64.40% in the final answer. Review changes the BEAM scores only slightly. The complete post-draft stage accounts for 25.03% of total input and output tokens across 812 executions. The paper provides qualitative examples of review failures. It does not report the frequencies of corrected errors and newly introduced errors. It also does not quantify the main causes of the AMA-Bench regression. This limits the explanation of harmful revisions and the justification for the additional computation.
3.The controlled comparisons use only part of the available test questions. For example, the BEAM 100K comparison includes only 100 of the 400 questions. Each question is evaluated only once. The paper does not report repeated runs or controlled comparisons on the full test sets. The stability of the small performance gains therefore remains unclear.
Questions for authors
1.The paper lacks matched experiments against established navigation methods. Its advantage under comparable inference budgets remains unclear. Can the authors add a comparison with MemWalker or NaviRAG using the same model and test questions? The comparison should use comparable inference budgets and report both answer quality and actual token use.
2.The paper lacks repeated-run validation of the small performance gains. The controlled comparisons also cover only part of the available test questions. Can the authors add independent runs on a fixed subset? These runs should include MemoryLOD and its baselines. Mean scores and variation across runs should be reported.
2 comments
Response to author
Thank you for the new NaviRAG experiment and the AMA-Bench analysis. The counts of corrected and newly introduced errors address my concern about answer review. The token costs also help explain its trade-offs.
The NaviRAG comparison still leaves the cost question open. Both methods have the same call limit, but MemoryLOD uses more input tokens at query time. Its lower construction cost is useful, but the overall trade-off needs further evaluation. Repeated runs are also missing. The confidence intervals do not measure variation across runs. The results therefore do not yet establish a stable advantage under comparable costs. For these reasons, I maintain my Weak Reject recommendation.
Thanks for the clear review. Here are answers to your two questions and to the point about the AMA-Bench regression after answer review.
Q1. Matched comparison with MemWalker or NaviRAG, same model and questions, with token use.
We ran NaviRAG, with the same model (Qwen3.8-27B with reasoning), the same 20 BEAM 100K questions, a model-call budget of 25 per question, and native scoring.
MemoryLOD wins on 7 questions, loses on 3 and ties on 10. Three of NaviRAG's 13 hierarchy constructions ran past a 4,000-call cap, so six questions count as failures; on the other 14 the scores are 27.62 vs 18.95. Model calls are matched, and output tokens at query time are about the same (0.90M vs 0.76M). MemoryLOD does read more input (2.45M vs 0.38M). That's the trade-off for using zero language-model calls for construction, where NaviRAG used 41.5M input and 81.9M output tokens. This is an adapted, capped port on a small set of questions, not a reproduction of the published results, and we didn't run MemWalker.
Against Hybrid RAG's single reader call, MemoryLOD uses 5.8–10.7x the tokens on the primary settings. Saving tokens isn't the goal. MemoryLOD aims to get as much relevant opened source evidence as possible to the reader within adjustable per-question resource limits (model-call budget, source-opening limit, evidence character budget). We report one setting of those limits, and we don't claim an advantage at equal cost.
Q2. Independent repeated runs on a fixed subset.
We couldn't finish repeated runs during the discussion period. What we do have is a paired 95% interval, resampled over source collections, for every matched difference, and we only claim the differences whose intervals exclude zero. Against Local Tree: BEAM 100K (+11.96 [+6.03, +18.72]), LongMemEval-V2 Medium (+50.00 [+32.61, +65.22]) and AMA-Bench (+8.40 [+1.59, +15.02]). Against Hybrid RAG: LongMemEval-V2 Medium (+45.65 [+28.26, +60.87]). We don't claim anything from the small differences against Hybrid RAG on BEAM 100K and AMA-Bench.
Weakness 2. Frequencies and causes of the AMA-Bench regression.
Looking at all 250 pairs of first draft and final answer, answer review fixes 7 answers and breaks 21 (154 stay correct and 68 stay incorrect), so accuracy goes from 70.00% to 64.40%. We used Codex (gpt-6-astra) to inspect the causes, shown in the table below:
All 7 fixes came after the navigator acquired new opened source evidence following the critic's request; 15 of the 21 regressions involved no new evidence. So review helps when it sends the navigator back for evidence, and hurts when it only raises the bar for support. On AMA-Bench the post-draft stage costs 24.77% of input and output tokens and isn't justified by accuracy. We'll add this analysis, report first-draft and final-answer scores side by side, and present answer review as a stage whose effect depends on the task.