- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Picorer, an agentic memory retrieval framework. The main design is to separate a revisable working state from an accumulated evidence ledger. The working state keeps currently established facts and missing information, and is used to guide the next search/read action. The evidence ledger keeps the source passages that have already been read and provides them to the final answer model.
The paper evaluates Picorer on MemoryAgentBench, BEAM, and LoCoMo. It also includes ablations and additional analysis on multi-hop retrieval, especially about cumulative evidence retention, complete supporting chains, and whether the model can correctly judge when the retrieved evidence is already sufficient.
Critical strengths
1. The paper has relatively extensive empirical analysis. Compared with many system papers that only report final benchmark numbers, this paper tries to analyze what happens during the retrieval process. In particular, the analysis on partial support versus complete support chain is useful for understanding the behavior of multi-hop retrieval.
2. The ablation studies reveal some concrete behavior of the system. The paper shows that the structured note mainly reduces repeated retrieval, while cumulative retention of previously read evidence has a much larger effect on multi-hop accuracy. I think this analysis is useful even though it also raises questions about the actual novelty of the proposed method.
Critical weaknesses
1. My main concern is the limited novelty of the method. The major principles used in Picorer, such as iterative retrieval, adapting the next retrieval based on intermediate results, and keeping information obtained from previous retrieval steps, have already been explored in prior multi-hop and agentic retrieval systems.
The most explicit new component seems to be the structured Established/Missing working note. However, according to the paper's own ablation, removing this note almost does not affect the final accuracy on the multi-hop fact consolidation task. Its major effect is reducing repeated source accesses. This makes the note look more like an engineering mechanism for avoiding redundant retrieval rather than a major algorithmic contribution.
The larger performance effect comes from keeping previously read evidence instead of only keeping the latest read. However, cumulative retention of previous evidence seems to be a rather natural design choice for an iterative multi-hop retrieval system. Therefore, I have difficulty seeing a sufficiently strong methodological novelty beyond engineering refinement of existing ideas.
2. The current experiments do not fully isolate where the gain comes from. The abstract emphasizes the comparison with HippoRAG-v2, but this comparison is not under a fully matched experimental setting compared with other controlled baselines.
Also, the w/o note ablation still keeps most of the Picorer framework, including the search/read loop, source information, budget information, and cumulative evidence ledger. Therefore, it does not answer a simpler question: whether a basic iterative retrieval system with query reformulation and cumulative passage retention can already achieve similar performance.
Without such a baseline, it is difficult to tell whether the gain is really from the proposed framework or mainly from standard iterative retrieval plus keeping all previously retrieved evidence.
Questions for authors
- Can the authors provide a controlled comparison with HippoRAG-v2 under the same backbone, corpus, retrieval budget, and evaluation setting, since the paper emphasizes improvement over HippoRAG-v2?
- Can the authors compare against a simpler baseline that only performs iterative query reformulation/search and cumulatively keeps the retrieved passages, without the Established/Missing note and other Picorer-specific state design? I think this baseline is important to understand how much of the gain actually comes from the proposed framework.