- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how an agent can find and organize useful memories for a current query after accumulating a large interaction history. The authors propose JAM, which preserves the full history, builds a hierarchical workspace, and trains a Researcher to perform multiple rounds of retrieval and reading to construct query-relevant context. The main contributions are a Just-In-Time memory paradigm and a two-component framework that implements it: a Memorizer organizes historical information offline, while a Researcher conducts query-driven research at runtime. The paper reports stronger overall results than the evaluated baselines on LoCoMo, LongMemEval, NarrativeQA, and long-context HotpotQA, and demonstrates cross-domain transfer on LongCodeQA.
Critical strengths
The paper offers a meaningful perspective on memory system design. Future queries are often difficult to anticipate, and compressing history in advance may discard details that later become important. JAM provides a sensible response to this information-loss problem and offers useful insights into how computation can be allocated between memory writing and query processing. The experiments provide reasonably strong evidence of the design’s practical effectiveness. The method achieves strong results across several memory and long-context tasks, and the ablation studies examine the contributions of training and hierarchical organization.
Critical weaknesses
However, the paper has two main limitations. First, it lacks a controlled comparison with a strong multi-turn agentic RAG system. JAM’s workflow is closely related to agentic RAG, but the experiments do not establish its advantage under the same backbone, training data, and comparable online compute budgets. Second, the analysis and evaluation of long-term memory behavior remain limited. The paper does not directly assess how growing histories and other changes during continuous memory updates affect system performance. As evaluated, JAM is closer to a historical information retrieval system than a demonstrated mechanism for continual memory management.
Questions for authors
I would appreciate clarification on the following points:
- Have the authors compared JAM with a multi-turn agentic RAG system using the same backbone, the same Memory-Gym training data, and a comparable online compute budget? If so, what were the results?
- Is the intended scope query-conditioned context construction over a given history, or long-term memory management during ongoing interactions?
- Which component produces the final answer used for evaluation? The descriptions in the main paper and appendix are somewhat unclear on this point.
I appreciate the authors' efforts in reporting the new experimental results. I am inclined to keep my original score, though I would have no objection if the AC recommends this paper for acceptance.