- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MemTact, which separates long-term LLM memory into two parts: Navigation Memory and Information Memory. The first part uses hierarchical semantic representations for retrieval, and the second part dynamically decides which evidence should be kept in the readable memory based on previous queries and recovery costs.
The paper also introduces a joint evidence placement optimization problem and shows that it can be solved exactly using minimum cut under certain assumptions.
Experiments compare MemTact with RAPTOR, A-MEM, and QueryLink on three benchmarks. MemTact achieves 61.52% overall accuracy, which is better than RAPTOR and A-MEM but worse than QueryLink. Its online token usage is lower than A-MEM and QueryLink but higher than RAPTOR.
Critical strengths
1. The paper considers both retrieval and reading costs.
I think separating findability and residency is a reasonable system design. The paper points out that information which can be retrieved does not necessarily need to stay in the memory that the LLM reads every time. This is a practical consideration for long-term memory systems, although I am not convinced this is a new idea.
2. Some experimental results are positive.
MemTact gets better accuracy than RAPTOR and A-MEM, and uses fewer online tokens than A-MEM and QueryLink. So the combined system does show some practical benefits in the evaluated settings.
Critical weaknesses
1. I have a hard time understanding what the actual methodological contribution is.
The paper presents Navigation Memory and Information Memory as two main contributions. But to me, Navigation Memory is basically a retriever with hierarchical semantic representations. Hierarchical retrieval, query expansion, and multi-view retrieval are already quite standard. It is difficult to see what is fundamentally new here.
For Information Memory, isn't this essentially a cache? The system keeps some frequently useful evidence, recovers information when needed, and updates what should remain in memory based on previous accesses. This is very similar to cache admission and replacement, or simply taking notes about what should be remembered for future queries.
I understand that the authors have some specific implementation choices, but why should these be considered methodological contributions rather than a combination of existing components? Most existing long-term memory systems already need retrieval and some form of memory update. Simply separating them into two modules does not seem sufficient.
2. The theoretical formulation and experimental claims are not convincing to me.
I find Section 3.4 particularly confusing. The paper introduces an exact optimization formulation using maximum-weight closure and minimum cut. But then the authors explicitly say that the actual online controller is a separate practical policy and does not solve this optimization problem.
Then why introduce this exact optimization formulation in the first place? If the implemented system neither solves it nor empirically validates its relevance, what is the actual contribution of this formulation? I cannot see how it connects to the algorithm being evaluated.
The experiments also do not convincingly support the claimed cost-optimal trade-off.
MemTact does not achieve the best accuracy, and it does not achieve the lowest online token usage either. Of course, I understand that a method does not need to be the best on every metric. It can be valuable if it achieves a better accuracy-cost trade-off.
But how do we know this is actually a better trade-off?
The paper only compares one operating point for each method. Maybe QueryLink with a smaller retrieval budget can achieve higher accuracy than MemTact at a similar token cost. Maybe another configuration of RAPTOR can reach a better accuracy-cost point. Without showing the accuracy-cost curves or comparing the Pareto frontiers, how can the paper conclude that MemTact provides a better balance?
Even the dynamic residency experiment does not clearly demonstrate cost savings. It improves accuracy, but also increases Reader token usage. The fact-reuse ablation shows token savings on a small selected set of target queries, but excludes the prefix-query cost required to build up the dynamic memory state from its reported target usage.
To me, the paper has not clearly demonstrated either the benefit of its optimization formulation or the claimed end-to-end cost advantage of its memory controller.
Questions for authors
- What exactly is new about the two proposed memory components?
- Why is the exact optimization formulation introduced if the system does not use it?
- How can the authors claim a good accuracy-cost trade-off based on only one operating point per method?
- Does dynamic residency actually save cumulative serving cost?