- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents MemGround, a benchmark for evaluating memory behavior in long-horizon interactive environments. The authors convert three existing games into structured environments representing linear, temporal, and graph-structured memory, where agents must progressively discover and organize information. The evaluation combines QA performance, exploration progress, temporal ordering, and diagnostic scores for memory retrieval and reasoning. Experiments across multiple LLMs and memory systems show that models struggle with temporal and structural reasoning, while external memory provides inconsistent improvements.
Critical strengths
- Evaluating memory through interactive environments rather than only static QA is an interesting direction.
- The data construction pipeline is a useful contribution. Representing gameplay as structured graphs with explicit memory dependencies enables traceable evaluation and could facilitate future benchmark development.
- The evaluation covers several strong models and established memory systems, including Mem0 and A-MEM. The authors also provide process-level metrics and human evaluation to better understand model failures.
Critical weaknesses
- The distinction between the three memory structures is not sufficiently motivated. Linear memory can be considered a special case of temporal memory, and both can be represented as graphs. It is unclear why the authors choose these three categories as the primary taxonomy.
- The selection of game environments appears somewhat ad-hoc. While games provide a reasonable setting for evaluating interactive memory, the paper does not sufficiently justify why these three games are representative. Moreover, each memory structure is associated with a different game, making it difficult to disentangle memory capabilities from game-specific mechanics and difficulty.
- The comparison with memory baselines is insufficient. The authors evaluate Mem0 and A-MEM, but do not consider native context management and compaction mechanisms available in frontier model APIs. Recent results on ARC-AGI-3 [1] suggest that retaining reasoning state and enabling native compaction can substantially improve performance on long-horizon interactive tasks. A baseline using the OpenAI Responses API with native compaction would strengthen the empirical evaluation.
- The benchmark is largely text-based, limiting its generalizability. Although one of the selected games is originally a visual novel, the evaluation converts gameplay into textual descriptions and structured records. This does not capture the challenges of multimodal memory in environments involving visual observations or spatial interactions.
References
[1] OpenAI. How enabling two settings tripled our scores on the ARC-AGI-3 benchmark. OpenAI 2026.
Questions for authors
Please refer to the weaknesses.
5 comments
Dear reviewer R3Ec,
Thanks for your time and effort in reviewing ICLR2027 submissions. The authors have submitted their responses to your review. Please take the time to read, raise any further comments, and discuss them with the authors. Thanks very much!
AC
Follow up about our response to Reviewer R3Ec
Dear Reviewer R3Ec,
We would like to kindly follow up regarding the detailed response we submitted addressing your insightful comments and questions on our paper. Your suggestions and feedback are invaluable and have greatly helped us improve our work. As the discussion period draws to a close, we remain available to provide any further clarification you may find helpful. Should all your concerns be addressed to your satisfaction, we would sincerely appreciate it if you could consider updating your assessment.
We sincerely appreciate the time and care you have devoted to reviewing our work.
Best regards,
The Authors
Response to Reviewer R3Ec (R1)
Thank you for your constructive review and insightful suggestions. We have clarified our memory behaviors and environment selection rationale, added a native context compaction baseline on GPT-5.2, and outlined our multimodal future work to address your concerns.
We do not view these patterns as mutually exclusive categories that cover all possible memory forms. Instead, we chose them because they are highly representative of real agentic interaction tasks, and they present a progressive increase in memory and reasoning difficulty: Linear Memory tests sequential retention and state consistency, Temporal Memory tests chronological reconstruction of dispersed events, and Graph-Structured Memory tests non-linear dependency exploration and multi-hop evidence integration. Together, they serve as progressive evaluation dimensions for long-horizon memory behavior.
Our core contribution is evaluating memory behavior in interactive tasks, with games serving simply as controlled, structured testbeds. Rather than claiming these specific games are universally representative, we matched three representative agentic memory patterns to games whose mechanics naturally instantiate them: TRPG for sequential tracking, No Case Should Remain Unsolved for temporal reconstruction, and Type Help for non-linear graph exploration. To verify that game mechanics and difficulty do not unfairly hinder evaluation, we provided clear rules, converted observations into uniform text records, and evaluated four unfamiliar human volunteers under the same rules (Appendix F.1). These human players achieved over 90% average QA Overall across scenarios, 100% MFU in both puzzle games, and 100% MFCO in No Case Should Remain Unsolved, confirming that humans without prior game experience can readily solve these tasks using their memory.
We evaluated a GPT-5.2 workflow with native context management and compaction. Tables R3-1 to R3-3 report its main results, six-dimensional QA scores, and MS/RS diagnostic metrics across all scenarios. Compared to the basic GPT-5.2 in-context setup, the native-context workflow improves QA Overall across all scenarios. These results suggest that native context management and compaction may help preserve useful task information, supporting more effective memory behavior by GPT-5.2 during long-horizon interactions. However, performance remains limited in tasks requiring heavy temporal reconstruction and cross-fragment linking (such as No Case Should Remain Unsolved), supporting our Section 5.2 finding that selecting and integrating dispersed evidence remains a key bottleneck.
Table R3-1. Main results of GPT-5.2 native-context workflow in MemGround.
Table R3-2. Six-dimensional QA evaluation of GPT-5.2 native-context workflow.
Table R3-3. Diagnostic protocol results (MS, RS, and FS) of GPT-5.2 native-context workflow.
MemGround currently focuses on the text-based domain, using textual observations and structured records to isolate and evaluate memory behavior and reasoning bottlenecks in long-horizon interactions. As acknowledged in Appendix H (Limitations), visual memory, spatial interaction, and multimodal environments are important aspects of agent memory, and extending MemGround toward multimodal environments remains a primary direction for our future work.