- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces MERA, a framework for answering questions over long conversation histories. It indexes short, self-contained facts extracted from the conversation where a lightweight planner generates retrieval queries using the question and accumulated evidence. The planner is trainable with GRPO using rewards based on newly recovered gold supporting turns. Experiments on LoCoMo and LongMemEval-S report improvements over searching recall and answer accuracy.
Critical strengths
- Well-motivated design that uses accumulated evidence to guide further searches while retaining access to the full memory.
Critical weaknesses
-
The work seems incremental, which does not clearly establish a clear advance over closely related contextual and structured-memory approaches.
-
Evaluation needs more justifications. It covers two datasets with dataset-specific training data, more specifications on the similarity between the training data and the evaluation data is needed. Generalization is insufficiently established, independent datasets outside the training distribution would strengthen the results. Table 1's results seem to come from different eval settings.
Questions for authors
- I would like to understand more on how similar the training data is as compared to the evaluation set in terms of conversation content, question types, and data generation procedures, and how potential overlaps are ruled out.
- To assess generalization, could the planner be trained using data from only one benchmark and evaluated on both? Excluding either LongMemEval-S or LoCoMo-derived training data would help distinguish cross-dataset transfer from gains due to benchmark-specific training.
- In Section 4.2, the footnote says "Results marked with ∗ are obtained from our own runs under the evaluation setup described above", which seems to come from a held-out test split of 101 questions based on Appendix B.1, are the results without * from evaluating on the same 101 questions or the 500 questions stated in 4.1? The accuracy numbers looks like they are evaluated on different settings.