- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces HIERAMEM, a "three-layer" hierarchical memory system to improve self-evolving agents. The motivation of work is clear that task-specific episodes may contain noise that couldn't not generalize to other tasks. So HIERAMEM proposes an Evidence Anchor module that helps transofrm episode level knowledge to re-usable abstract knowledge. The authors present comprehensive empirical evidence that HIERAMEM can outperform previous Memory systems with some analysis showing more in-depth insights on how their system works.
However, the presentation is very unclear. I encourage the authors to significantly simplify and clarify the methods for the reader to easily follow. I think the author should just highlight the key differentiator in their work with examples or with better illustrations and keep other parts as straightforward as possible. For these reason, I think the paper requies another revision.
Critical strengths
- As mentioned in the paper summary above, the motivation of the work is very clear.
- The overall method is intuitive to address the issues from "prior" works.
- Comprehensive experiments definitely add points.
Critical weaknesses
The biggest weakness of the paper is that the authors try to over-complicate the proposed method by throwing in too many math notations in the method section. This backfires. Compared with existing solutions (not just from academic papers, but also industry level products) to construct generalizible memory, the proposed method appears to be similar. This is acceptable if authors can clearly demonstrate the details and key differentiator of the method. However, the current narrative adds to the confusion. Here are a few concrete examples.
- Eq(4): The choice of using goal is unclear here and the goal wasn't explain previously. This can be easiy illustrated with an example.
- Eq(5): z and u are very abstract and z is defined with 4 different operations again, each of them are abract.
- Eq(6) + Eq(8): P are not explained - though later I understand these are prompts. Also, (d, S, m) is defined by m and A, but A is defined with m again. The logic is convoluted. Simple example or illustration works much better.
Another weakness (which is acceptable) is that the tested benchmarks are verifiable with clear outcomes. However, agents nowadays are moving towards non-verifiable real-world tasks. So good to test if the proposed method can generalize there.
Questions for authors
- Do you think the Evidence Anchor is another level or just a bridge between the specific knowledge to the more generalizible knowledge? If you think it is a level of memory, can downstream tasks retrieve this layer directly?
- L172-173, what exactly is cold-start information and how do you derive memory from it alone?
Thank you for the detailed response and additional experiments. The Direct SYM Retrieval comparison provides more direct evidence for the value of anchor-mediated retrieval, and the success-only, failure-only, and contrastive comparisons strengthen the evidence for memory formation. Together with the added variability statistics and protocol clarifications, these results make the contribution of the evidence-anchor layer more convincing. I will raise my rating to 3.