- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the mismatch between "demands" and "evidence" within long-term memory systems, distinguishing between "retrieval relevance" and "evidential sufficiency." It introduces a diagnostic framework called MemPath, which examines the accessibility of source information, the completeness of demands, and the success of answer retrieval under the condition of "ideal evidence intervention"—a scenario assuming the acquisition of perfect evidence. Building on this analysis, the BiDE-Mem framework incorporates "Typed Question Demands" to guide memory access and evidence construction, ensuring the alignment of demands and evidence prior to generation. Experimental results on the LoCoMo and LongMemEval datasets demonstrate the superior performance of this approach; furthermore, the study validates its effectiveness through component ablation experiments and analysis at the evidence level.
Critical strengths
MemPath goes beyond focusing solely on end-to-end accuracy; it also examines evidence accessibility, requirement coverage, and answer recovery via interventions, thereby offering valuable insights into the issue of memory failure.
On the LoCoMo task, BiDE-Mem achieves a significant boost in accuracy by sharing an answer-generation model, while simultaneously requiring fewer input tokens for the reader compared to several strong baseline models.
Critical weaknesses
-
Regarding Figure 2. Figure 2a shows that, within a diagnostic subset of 250 questions, ReadAgent achieved a "demand-complete" status for 41.2% of the questions (i.e., 103 questions). However, Figure 2b identifies 164 cases as "covered but wrong." If both subfigures utilize the same subset and the same criteria for "completeness," the latter figure should not exceed the total number of "demand-complete" questions. The authors are requested to clarify the sample sets, denominators, and definitions used for both subfigures, and to correct the data in the figures if necessary.
-
Regarding the transparency of the human review process. Appendix B.2 mentions the creation of a "human-reviewed demand annotation set," consisting of 250 LoCoMo questions and 425 "question demands." The authors should specify the number and qualifications of the annotators/reviewers, the detailed guidelines provided for annotation and review, the mechanisms for resolving disagreements, and—if applicable—the inter-annotator agreement metrics. Furthermore, please clarify whether the human review was limited to this diagnostic subset or if it encompassed all 2,517 demands.
Questions for authors
A minor point regarding source-based demand coverage. Table 10 defines "supporting sources" as source IDs; Appendix B.3 defines as the set of annotated supporting sources for a given demand , using as the criterion for "fully supported." Consequently, represents source identifiers rather than the actual content of the evidence. When constructing , did the authors merely verify the presence of the source IDs, or did they also confirm that the information required to satisfy the demand was indeed preserved within the evaluation context? This distinction is critical, especially in the context of "compressed memories"—retaining a source ID does not necessarily imply that the required evidentiary content has been preserved.
2 comments
Thank you for the detailed response and corrections. However, I am not convinced that my main concerns have been resolved, so I will maintain my original score.
Diagnostic Consistency, Human Review, and Evidence Coverage
We thank the reviewer for identifying the Figure 2 inconsistency and raising questions about annotation and source coverage. We have corrected the figure and clarified the statistics, human review, and evidence checks.
1. Figure 2: Error Correction and Statistical Definitions
The reviewer is correct about the inconsistency. In the original Figure 2b, the value 164 for ReadAgent was its total number of initially incorrect answers on the 250-question subset, but was placed under Covered but wrong. It did not represent 164 demand-complete incorrect cases. Figure 2a's value, 103/250 (41.20%), was the question-level demand-complete count and remains valid. The corrected Figure 2 replaces the error-routing panel with (a) question-level source-grounded demand completeness and (b) oracle evidence-intervention recovery.
The 250 questions / 425 demands across six systems yield 1,500 system–question cases, including 915 initially incorrect. The aggregate intervention records include 869 valid cases per setting. Evidence-only replaces the original context with reviewed supporting evidence; Combined integrates that evidence with the existing context. Both keep the Reader and answer-generation procedure unchanged. Evidence-only recovers 404/869 (46.49%) and Combined recovers 476/869 (54.78%).
Source-grounded completeness is measured over questions per system, while intervention recovery uses initially incorrect system–question cases. The aggregate records do not establish identical case membership across the two interventions or item-level exclusion reasons for the remaining 46 errors. We report these rates descriptively, not as a matched-pair significance comparison. The revised Section 2.2 and Appendix B.3 specify the definitions and denominators.
2. Demand Annotation and Human Review
We used an LLM-assisted annotation followed by human cross-review. The LLM drafted atomic Question Demands from the question, reference answer, and supporting sources. Two PhD researchers with relevant expertise participated in checking (i) whether the demands covered all necessary answer requirements, (ii) whether decomposition was atomic and the Fact, Relation, Temporal, State, Constraint types were applied consistently, and (iii) whether each demand was grounded in its annotated supporting sources without unsupported inferences.
The two inventories serve different purposes: 250 questions / 425 reviewed demands form the human-reviewed MemPath subset for cross-system diagnosis; 1,540 questions / 2,517 annotated demands form the full-scale offline BiDE-Mem coverage analysis. The latter is not claimed to have received independent dual human review. The retained records lack independent initial label sets, per-reviewer workloads, and complete disagreement-resolution records; hence, no inter-annotator agreement statistic is inferred (Appendix B.2).
Gold demands, reference answers, and annotated sources are offline evaluation inputs, not supplied to online QueryFrame, retrieval, evidence assembly, or answering.
3. Source-grounded Coverage in Compressed Memories
We agree that retaining a source ID does not prove that the required content survives compression. For demand di, Ai denotes its annotated supporting-source set, and Si(EQ) denotes the sources whose demand-relevant information is retained in the observed evidence context. Our all-support criterion remains Ai⊆Si(EQ); this definition is shared across memory representations.
For uncompressed, source-preserving records, source IDs are matched to the accessible original content. For compressed summaries or structured memories, we additionally inspect the retained representation through demand-indexed semantic mapping: it must preserve relevant entities, relation direction, temporal scope, polarity/state changes, or constraints as required by di.
For example, a summary retaining a source ID but dropping a required date does not count as support for that Temporal demand, even if it preserves information supporting a different demand. Thus, semantic preservation determines whether a source counts as retained for that demand; it does not replace or relax the source-grounded all-support rule.
Source-grounded completeness measures annotated supporting-information availability, not guaranteed answer correctness. It is distinct from the Reader's ability to use that evidence; the oracle interventions provide complementary evidence about answer recovery. These clarifications appear in Section 2.2 and Appendix B.3.
We appreciate the reviewer's careful reading, which helped us correct Figure 2 and make the annotation and evidence-coverage protocols more transparent.