acceptodds
Under review as a conference paper at ICLR 2027

MemGround: Evaluating Memory Behavior of Large Language Models in Long-Horizon Gamified Scenarios

Abstract

Long-term memory in large language models (LLMs) is typically evaluated by asking whether a model can retrieve and reason over information from a fixed interaction history. However, real interactive agents face a fundamentally different challenge: relevant information is progressively revealed, world states continuously evolve, and useful evidence must be discovered, organized, updated, and reused over extended interactions. This suggests evaluating long-term memory not merely as information retention, but as memory behavior—the ability to organize and apply accumulated evidence throughout interaction. We introduce MemGround, a benchmark that operationalizes this perspective through long-horizon gamified interactive environments. MemGround assesses memory behavior across three memory-structure patterns: Linear Memory, Temporal Memory, and Graph-Structured Memory. To evaluate memory behavior comprehensively, we design a two-stage framework that combines end-to-end behavioral evaluation and process-level diagnosis. The end-to-end evaluation examines whether models can use accumulated information to answer questions correctly, make sustained progress during interaction, recover temporal organization among fragmented memories, and navigate structural dependencies coherently. We quantify these behaviors using Question-Answer Overall (QA Overall), Memory Fragments Unlocked (MFU), Memory Fragments with Correct Order (MFCO), and Exploration Trajectory Diagram (ETD). Beyond these observable outcomes, we further introduce Memory Score (MS) and Reasoning Score (RS) to identify whether failures stem from insufficient evidence retrieval or from incorrect reasoning over retrieved evidence. Experiments across state-of-the-art LLMs and memory-augmented agents reveal persistent limitations in dynamic state tracking, temporal association, and evidence-grounded reasoning over long-horizon interactions. We hope MemGround can serve as a systematic testbed for advancing the memory capabilities of future LLMs and agents.

Primary area:
datasets and benchmarks
Keywords:
Long-Horizon Memory Behavior, Benchmark, Interactive Gamified Scenario

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_R3EcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MemGround, a benchmark for evaluating memory behavior in long-horizon interactive environments. The authors convert three existing games into structured environments representing linear, temporal, and graph-structured memory, where agents must progressively discover and organize information. The evaluation combines QA performance, exploration progress, temporal ordering, and diagnostic scores for memory retrieval and reasoning. Experiments across multiple LLMs and memory systems show that models struggle with temporal and structural reasoning, while external memory provides inconsistent improvements.

Critical strengths
  1. Evaluating memory through interactive environments rather than only static QA is an interesting direction.
  2. The data construction pipeline is a useful contribution. Representing gameplay as structured graphs with explicit memory dependencies enables traceable evaluation and could facilitate future benchmark development.
  3. The evaluation covers several strong models and established memory systems, including Mem0 and A-MEM. The authors also provide process-level metrics and human evaluation to better understand model failures.
Critical weaknesses
  1. The distinction between the three memory structures is not sufficiently motivated. Linear memory can be considered a special case of temporal memory, and both can be represented as graphs. It is unclear why the authors choose these three categories as the primary taxonomy.
  2. The selection of game environments appears somewhat ad-hoc. While games provide a reasonable setting for evaluating interactive memory, the paper does not sufficiently justify why these three games are representative. Moreover, each memory structure is associated with a different game, making it difficult to disentangle memory capabilities from game-specific mechanics and difficulty.
  3. The comparison with memory baselines is insufficient. The authors evaluate Mem0 and A-MEM, but do not consider native context management and compaction mechanisms available in frontier model APIs. Recent results on ARC-AGI-3 [1] suggest that retaining reasoning state and enabling native compaction can substantially improve performance on long-horizon interactive tasks. A baseline using the OpenAI Responses API with native compaction would strengthen the empirical evaluation.
  4. The benchmark is largely text-based, limiting its generalizability. Although one of the selected games is originally a visual novel, the evaluation converts gameplay into textual descriptions and structured records. This does not capture the challenges of multimodal memory in environments involving visual observations or spatial interactions.

References

[1] OpenAI. How enabling two settings tripled our scores on the ARC-AGI-3 benchmark. OpenAI 2026.

Questions for authors

Please refer to the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer R3Ec (R1)

Thank you for your constructive review and insightful suggestions. We have clarified our memory behaviors and environment selection rationale, added a native context compaction baseline on GPT-5.2, and outlined our multimodal future work to address your concerns.

W1. Distinction between linear, temporal, and graph-structured memory.

We do not view these patterns as mutually exclusive categories that cover all possible memory forms. Instead, we chose them because they are highly representative of real agentic interaction tasks, and they present a progressive increase in memory and reasoning difficulty: Linear Memory tests sequential retention and state consistency, Temporal Memory tests chronological reconstruction of dispersed events, and Graph-Structured Memory tests non-linear dependency exploration and multi-hop evidence integration. Together, they serve as progressive evaluation dimensions for long-horizon memory behavior.

W2. Rationale for environment selection and game mechanics.

Our core contribution is evaluating memory behavior in interactive tasks, with games serving simply as controlled, structured testbeds. Rather than claiming these specific games are universally representative, we matched three representative agentic memory patterns to games whose mechanics naturally instantiate them: TRPG for sequential tracking, No Case Should Remain Unsolved for temporal reconstruction, and Type Help for non-linear graph exploration. To verify that game mechanics and difficulty do not unfairly hinder evaluation, we provided clear rules, converted observations into uniform text records, and evaluated four unfamiliar human volunteers under the same rules (Appendix F.1). These human players achieved over 90% average QA Overall across scenarios, 100% MFU in both puzzle games, and 100% MFCO in No Case Should Remain Unsolved, confirming that humans without prior game experience can readily solve these tasks using their memory.

W3. Native context management as an additional baseline.

We evaluated a GPT-5.2 workflow with native context management and compaction. Tables R3-1 to R3-3 report its main results, six-dimensional QA scores, and MS/RS diagnostic metrics across all scenarios. Compared to the basic GPT-5.2 in-context setup, the native-context workflow improves QA Overall across all scenarios. These results suggest that native context management and compaction may help preserve useful task information, supporting more effective memory behavior by GPT-5.2 during long-horizon interactions. However, performance remains limited in tasks requiring heavy temporal reconstruction and cross-fragment linking (such as No Case Should Remain Unsolved), supporting our Section 5.2 finding that selecting and integrating dispersed evidence remains a key bottleneck.

Table R3-1. Main results of GPT-5.2 native-context workflow in MemGround.

ModelMethodTRPG (QA Overall)No Case Should Remain Unsolved (QA Overall)No Case Should Remain Unsolved (MFCO)No Case Should Remain Unsolved (MFU)Type Help (QA Overall)Type Help (MFU)
GPT-5.2Native Compaction(in-context)66.8334.024.5587.0465.7912.73

Table R3-2. Six-dimensional QA evaluation of GPT-5.2 native-context workflow.

ModelScenarioAcc. (%)Read. (%)Comp. (%)Dep. (%)Inst. (%)Cit. (%)Overall (%)
GPT-5.2TRPG - Cold Wind Howling71.4367.5870.0070.4097.8096.7072.49
TRPG - Spring Snow Incident58.0266.4158.1858.5199.2498.4764.72
TRPG - Terror on the Orient Express57.7563.3758.4457.7895.1995.1963.29
No Case Should Remain Unsolved22.2248.1514.2912.5085.1992.5934.02
Type Help60.8765.2860.9360.8297.0796.5865.79

Table R3-3. Diagnostic protocol results (MS, RS, and FS) of GPT-5.2 native-context workflow.

ModelScenarioFS (%)MS (%)RS (%)
GPT-5.2TRPG - Cold Wind Howling69.368.769.8
TRPG - Spring Snow Incident66.866.567.0
TRPG - Terror on the Orient Express63.563.463.6
No Case Should Remain Unsolved51.453.149.6
Type Help33.027.638.5

W4. Scope of the text-based evaluation.

MemGround currently focuses on the text-based domain, using textual observations and structured records to isolate and evaluate memory behavior and reasoning bottlenecks in long-horizon interactions. As acknowledged in Appendix H (Limitations), visual memory, spatial interaction, and multimodal environments are important aspects of agent memory, and extending MemGround toward multimodal environments remains a primary direction for our future work.

Dear reviewer R3Ec,

Thanks for your time and effort in reviewing ICLR2027 submissions. The authors have submitted their responses to your review. Please take the time to read, raise any further comments, and discuss them with the authors. Thanks very much!

AC

AuthorsOpenReview ↗

Follow up about our response to Reviewer R3Ec

Dear Reviewer R3Ec,

We would like to kindly follow up regarding the detailed response we submitted addressing your insightful comments and questions on our paper. Your suggestions and feedback are invaluable and have greatly helped us improve our work. As the discussion period draws to a close, we remain available to provide any further clarification you may find helpful. Should all your concerns be addressed to your satisfaction, we would sincerely appreciate it if you could consider updating your assessment.

We sincerely appreciate the time and care you have devoted to reviewing our work.

Best regards,

The Authors

Reviewer_74irOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

MemGround evaluates how language models discover, organize, update, and use information during extended interaction. It instantiates three proposed memory structures through three TRPG campaigns and two puzzle games. Linear Memory concerns sequential state tracking. Temporal Memory requires reconstructing event order. Graph Structured Memory requires navigating dependencies among files and clues. The benchmark contains 6,179 calibrated scenario units and 482 open ended questions. Evaluation combines answer quality, unlocked memory fragments, temporal ordering, and exploration trajectories. A second diagnostic protocol uses Memory Score and Reasoning Score to attribute failures to retrieval or reasoning. The experiments compare five language models and two external memory systems. The results show large human to model gaps and substantial variation across models, scenarios, and memory methods.

Critical strengths
  1. The two puzzle environments expose a meaningful failure mode that static question answering benchmarks rarely measure. In No Case Should Remain Unsolved, GPT 5.2 unlocks 81.48 percent of the fragments but obtains only 15.91 percent MFCO. All three Qwen configurations obtain zero MFCO. Four human participants reach 100 percent MFU and MFCO under the controlled protocol. This supports the claim that retrieving information and organizing it are distinct difficulties.

  2. The annotation pipeline is traceable and carefully documented. Each question records its required evidence and reasoning path. Scenario calibration uses two independent annotators followed by adjudication, with an overall Cohen's κ of 0.82. The paper also reports human evaluations of the model outputs rather than relying only on one automated judge.

Critical weaknesses
  1. The main metrics do not yet provide a reliable basis for comparing systems. Type Help MFU appears to include fragments unlocked through benchmark hints. The counts in Figure 6 reproduce Table 2 when self unlocked and hint unlocked fragments are added together. For Gemini, 46 self unlocked fragments plus 10 hint unlocked fragments produce 50.91 percent under a denominator of 110. This conflicts with the claim that assisted and autonomous progress remain separate. The denominator also differs from the 104 final Type Help units reported in Table 8. QA Overall is similarly unstable across judges. On No Case Should Remain Unsolved, GLM 5.1 reports that Mem0 reduces GPT 5.2 from 22.3 to 21.2. Human judges instead report an increase from 32.1 to 44.8. The best configuration changes from Claude Opus under GLM 5.1 to GPT 5.2 with A-MEM under human evaluation. The paper provides no item level agreement, correlation, or uncertainty analysis. Comp and Dep contribute 30 percent of QA Overall but their exact computation is not specified.

  2. The benchmark does not isolate memory structure from scenario and evaluation artifacts. Each proposed structure corresponds to only one game type. Differences attributed to linear, temporal, or graph memory may instead arise from game mechanics, translation quality, action space, or prompt design. TRPG is also based on summarizing recorded logs rather than interacting with the game, although the full benchmark is presented as interactive. These three campaigns contribute 409 of the 482 questions. The MS and RS protocol does not convincingly separate retrieval from reasoning. For in context systems, the Retrieved set used to compute evidence precision and recall is not defined. Both semantic retrieval quality and reasoning quality rely heavily on post hoc judging of the same answer and rationale. No oracle evidence intervention tests whether reasoning succeeds once retrieval is fixed. The resulting scores are often close, and only 1.3 percent of outputs are classified as Memory Weak alone.

Questions for authors
  1. Does Type Help MFU include fragments unlocked through hints? Why is the implied denominator 110 when Table 8 reports 104 final units? Please also reconcile the different descriptions of the hint mechanism in Section 4.2 and Appendix E.2.2.

  2. How are Comp and Dep computed? How many human judges evaluated each output, were they blind to system identity, and what is the item level agreement between human and GLM 5.1 scores?

  3. How many independent runs were performed for each configuration? Please provide temperature, seed, context limits, compression thresholds, retrieval settings, and uncertainty estimates. This is especially important for the scenarios containing only 27 and 46 questions.

  4. How is the Retrieved set defined for in context systems when computing MS? What thresholds produce the four error categories? Is there an analysis where every model receives the oracle evidence so that reasoning can be measured independently of retrieval?

  5. Can the authors provide deeper insights that hold across models? For example, why does high MFU consistently fail to produce high MFCO, and why does external memory help Qwen in Type Help while A-MEM reduces GPT 5.2 performance there? A cross model error analysis would be more useful than the current conclusion that compatibility varies.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 74ir (R1)

Thank you for your careful review and detailed questions. We have added no-hint exploration runs, agreement analyses involving five human raters, three-run variance statistics, full oracle-evidence evaluations, and cross-model error analyses to address your concerns.

W1 / Q1. Purpose of MFU, node counts, and hint mechanism.

We designed Memory Fragments Unlocked (MFU) to measure overall information coverage by the end of a run, showing how well a model sustains exploration progress. In Figure 6 (Appendix E.2.2), we already use different node colors to clearly distinguish self-unlocked fragments from hint-assisted ones. The counts of 104 and 110 refer to different node sets: the full Type Help graph has 110 nodes in total, including 6 initial nodes available at the start and 104 exploration-dependent fragments. Appendix Table 8 reports the 104 exploration fragments to elaborate the collection procedure of exploration-dependent fragments, while MFU in Table 2 uses all 110 nodes (e.g., for Gemini, MFU = (6 + 40 + 10) / 110 × 100% = 50.91%).

Hints are used to prevent early deadlocks from stopping the game too soon so that long-horizon memory retention can be evaluated. Thank you for pointing out the wording difference between Section 4.2 and Appendix E.2.2. In our actual Type Help pipeline, after 50 consecutive steps without discovering a new file, the system places the target filename into the readable directory, and the model must still actively choose and read that file to access its content. Table R2-1 reports results with hints disabled.

We have clarified these descriptions in both the main text and appendix of the revised paper.

Table R2-1. Type Help results with hints completely disabled.

ModelMethodQA Overall (%)No-hint MFU (%)
Gemini-3-Pro-PreviewIn-context21.9523.64
DeepSeek-V3.2In-context18.5911.82
Claude-Opus-4.6In-context16.1522.73
GPT-5.2In-context13.3012.73
Mem024.058.18
A-MEM12.2511.82
Qwen3-VL-32B-InstructIn-context5.507.27
Mem06.097.27
A-MEM10.878.18

W1 / Q2. Definitions of Comp. and Dep., human setup, and judge agreement.

Comp. focuses on information integration, while Dep. summarizes performance by reasoning depth. Their exact computations are defined as follows:

  • Comp. = Number of CONSISTENT answers in Contradiction (category 2) and Comparison (category 5) / Total number of questions in categories 2 and 5 ×100%\times 100\%.
  • Dep. = (Surface accuracy + Character accuracy + Cross-section accuracy) / 3. Dep. averages the accuracy across the three depth levels to give each level equal weight, preventing levels with larger item counts from dominating the metric.

Including these two terms in the weighted overall score highlights differences in information integration and reasoning depth across models. The exact formulas for Comp. and Dep. have been added to Appendix E.1.

For human evaluation, five human raters independently assessed the QA outputs under blind conditions using the same rubric, and their mean scores are reported in Appendix E.3. Table R2-2 summarizes the configuration-level agreement between the five-rater mean scores and GLM-5.1 across all QA dimensions and diagnostic metrics. Across all scenarios, Pearson rr ranges from 0.784 to 0.899 across the six QA dimensions and exceeds 0.80 for both MS and RS, showing a positive correlation and consistent evaluation trends. Details have been updated in Appendix E.3.

Table R2-2. configuration-level agreement between human average ratings and GLM-5.1 across QA dimensions and diagnostic metrics.

MetricPearson rrQuadratic Weighted κ\kappa
Acc.0.7960.531
Read.0.8820.654
Comp.0.7840.555
Dep.0.8620.565
Inst.0.8981.000
Cit.0.8990.902
QA Overall0.8890.673
MS (Memory Score)0.8050.607
RS (Reasoning Score)0.8280.556
FS (Final Score)0.8220.527
AuthorsOpenReview ↗

Response to Reviewer 74ir (R2)

Q3. Repeated runs and experimental reproducibility.

For the QA evaluation, we ran each model configuration independently three times. As detailed in Table R2-3, the sample standard deviations of QA Overall range from 0.37 to 3.51 points, indicating low variance across runs.

In our setup, we set temperature to 0.2 and limit the context window to 65,000 tokens. Under context compression, once an interaction reaches 100 rounds, the model summarizes the first 80 rounds of history. External memory agents (Mem0 and A-MEM) use their native retrieval interfaces and retain the top 5 memory items per query, while step budgets and early-stopping rules follow Section 4.2. (Details are revised in Section 4.2)

Table R2-3. QA Overall mean and standard deviation across three independent runs.

ModelMethodTRPGNo Case Should Remain UnsolvedType Help
Gemini-3-Pro-PreviewIn-context20.1 ± 1.6926.4 ± 3.2128.1 ± 2.02
DeepSeek-V3.2In-context23.3 ± 2.2533.2 ± 2.0026.2 ± 2.99
Claude-Opus-4.6In-context25.1 ± 0.3741.3 ± 2.7022.6 ± 2.92
GPT-5.2In-context37.5 ± 1.0722.3 ± 3.2423.2 ± 3.51
Mem038.0 ± 1.7621.2 ± 3.2427.9 ± 2.30
A-MEM39.1 ± 1.1733.8 ± 2.6920.8 ± 2.56
Qwen3-VL-32B-InstructIn-context22.6 ± 1.6611.6 ± 2.5310.1 ± 2.33
Mem025.7 ± 0.598.9 ± 1.6713.8 ± 2.62
A-MEM25.2 ± 1.485.6 ± 1.3016.0 ± 3.15

W2 / Q4. Evidence selection, diagnostic thresholds, and oracle evaluation.

For in-context systems, we infer the evidence used from the models’ reasoning and answers and map it to the benchmark fragments to form the Retrieved set, comparing it against gold evidence for precision and recall.

In our diagnostic protocol, Correct answers are labeled Correct. For incorrect answers, an answer is labeled BOTH_WEAK when both MS and RS are below 60, MEMORY_WEAK when MS is below 60 and RS is 60 or above, and REASONING_WEAK when MS is 60 or above and RS is below 60. Incorrect answers with both MS and RS at 60 or above are also assigned to REASONING_WEAK as a fallback. We supplement these details in Appendices E.6.1 and E.6.2.

Following your suggestion, we also conducted oracle-evidence experiments where models directly receive annotated supporting evidence. As shown in Table R2-4, most metrics improve substantially once retrieval barriers are removed, indicating that acquiring and selecting evidence is a key bottleneck in the workflow. (Note: Since DeepSeek-V3.2 and Gemini-3-Pro-Preview has been discontinued, we use DeepSeek-V4.1-Flash and Gemini-3.1-Pro-Preview as substitutes and provide their full results in Table R2-5/6/7)

AuthorsOpenReview ↗

Response to Reviewer 74ir (R3)

Table R2-4. Oracle-evidence results across five models (in-context) and five scenarios (six-dimensional evaluation).

ModelScenarioOverall (%)Acc. (%)Inst. (%)Cit. (%)Read. (%)Comp. (%)Dep. (%)
GPT-5.2TRPG - Cold Wind Howling72.4362.6489.0185.7188.4660.0062.42
TRPG - Spring Snow Incident75.4961.8395.4290.8495.0465.4562.00
TRPG - Terror on the Orient Express74.2564.1789.8489.3090.1163.6463.11
No Case Should Remain Unsolved80.5366.6796.30100.00100.0071.4366.67
Type Help80.9969.5795.6589.1397.8372.2271.33
Claude-Opus-4.6TRPG - Cold Wind Howling79.8668.13100.0090.1198.3567.5068.56
TRPG - Spring Snow Incident79.5167.9496.1893.1397.3369.0967.36
TRPG - Terror on the Orient Express79.0168.8297.8595.7098.9259.2167.50
No Case Should Remain Unsolved82.2477.7888.89100.00100.0071.4358.33
Type Help82.5371.7482.6191.3097.8383.3369.78
Gemini-3.1-Pro-PreviewTRPG - Cold Wind Howling73.1158.24100.0091.2198.9052.5056.85
TRPG - Spring Snow Incident73.4456.4998.4790.8499.6258.1856.09
TRPG - Terror on the Orient Express71.8054.55100.0093.0599.2053.2553.60
No Case Should Remain Unsolved87.0585.1985.19100.00100.0085.7162.50
Type Help79.7769.5773.9191.3098.9172.2267.52
DeepSeek-V4.1-FlashTRPG - Cold Wind Howling77.6862.64100.0094.51100.0065.0062.77
TRPG - Spring Snow Incident75.7059.54100.0090.0899.2463.6460.12
TRPG - Terror on the Orient Express77.1965.2498.9397.3398.9358.4462.39
No Case Should Remain Unsolved89.7985.19100.0096.30100.0085.7177.09
Type Help81.2271.7486.9693.4898.9172.2267.80
Qwen3-VL-32B-InstructTRPG - Cold Wind Howling69.1052.75100.0092.3194.5150.0052.05
TRPG - Spring Snow Incident63.8443.5196.1888.5593.1345.4545.29
TRPG - Terror on the Orient Express71.1458.2997.3389.3093.3251.9556.87
No Case Should Remain Unsolved72.8666.6781.4892.5996.3064.2937.50
Type Help79.7669.5776.0984.7895.6577.7869.89

Table R2-5. Main results of the replacement models in the in-context configuration.

ModelTRPG (QA Overall)No Case Should Remain Unsolved (QA Overall)No Case Should Remain Unsolved (MFCO)No Case Should Remain Unsolved (MFU)Type Help (QA Overall)Type Help (MFU)
DeepSeek-V4.1-Flash54.1353.8825.0083.3349.7290.00
Gemini-3.1-Pro-Preview29.6149.7475.0090.7420.7069.09

Table R2-6. Six-dimensional QA results of the replacement models in the in-context configuration.

ModelScenarioOverall (%)Acc. (%)Inst. (%)Cit. (%)Read. (%)Comp. (%)Dep. (%)
DeepSeek-V4.1-FlashTRPG - Cold Wind Howling52.6346.1596.7092.3152.7545.0045.04
TRPG - Spring Snow Incident57.7551.1590.8498.4762.2145.4549.76
TRPG - Terror on the Orient Express52.0045.9996.2687.7052.1444.1644.96
No Case Should Remain Unsolved53.8844.4492.5992.5970.3742.8625.00
Type Help49.7236.9689.1386.9658.7044.4437.00
Gemini-3.1-Pro-PreviewTRPG - Cold Wind Howling26.8223.08100.0040.6624.1815.0022.40
TRPG - Spring Snow Incident38.8533.5998.4769.4739.6925.4530.98
TRPG - Terror on the Orient Express23.1719.7997.8634.7617.6516.8818.48
No Case Should Remain Unsolved49.7437.0496.3096.3064.8142.8620.84
Type Help20.7015.2297.8332.6118.4811.1116.04
Reviewer_ranqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors identified a critical issue for interactive agents that, relevant information is progressively revealed, world states continuously evolve, and useful evidence must be discovered, organized, updated, and reused over extended interactions. Yet, LLMs are still usually only asked to retrieve fixed, existing knowledge. So they introduced this benchmark, MemGround, with long-horizon gamified interactive environments.

Critical strengths
  1. The motivation is strong, as agent memory is still a critical open problem.

  2. The categories of memory structures, namely linear, temporal, and graph-structured memories, make sense and are reasonably inclusive of memory types being used by agents.

  3. Evaluation is comprehensive.

Critical weaknesses

My main concern is the diagnostics. Poor performance may have various reasons, including weakness in reasoning abilities. I didn't find justifications that poor/good performances in many tasks are indeed or likely due to poor memory, rather than the underlying models' internal abilities.

Questions for authors

Please see Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ranq (R1)

Thank you for your thoughtful review and constructive feedback. We have conducted additional oracle-evidence experiments across all five models and scenarios to address your concern regarding intrinsic reasoning versus memory limitations.

W1. Disentangling memory limitations from underlying reasoning ability.

In Section 5, we introduce a diagnostic protocol with Memory Score (MS) and Reasoning Score (RS) to separate complex long-horizon behavior into two components: MS measures whether a model can locate and retrieve the required evidence, while RS independently evaluates its reasoning ability using that evidence. To further remove context noise, retrieval limits, and game rules, we conducted an oracle-evidence experiment where models bypass exploration and directly receive the annotated supporting evidence for every question. As shown in Table R1-1, directly providing evidence improves overall performance across all models. However, even without retrieval barriers, models still fall short of perfect scores, indicating that they often struggle with reasoning and integrating evidence even when given the right information. This aligns with our main diagnostics and error analysis, confirming that intrinsic reasoning limits remain a key bottleneck alongside memory. (Note: Since DeepSeek-V3.2 and Gemini-3-Pro-Preview has been discontinued, we use DeepSeek-V4.1-Flash and Gemini-3.1-Pro-Preview as substitutes and provide their full results in Table R1-2/3/4. )

Table R1-1. Oracle-evidence results across five models (in-context) and five scenarios (six-dimensional evaluation).

ModelScenarioOverall (%)Acc. (%)Inst. (%)Cit. (%)Read. (%)Comp. (%)Dep. (%)
GPT-5.2TRPG - Cold Wind Howling72.4362.6489.0185.7188.4660.0062.42
TRPG - Spring Snow Incident75.4961.8395.4290.8495.0465.4562.00
TRPG - Terror on the Orient Express74.2564.1789.8489.3090.1163.6463.11
No Case Should Remain Unsolved80.5366.6796.30100.00100.0071.4366.67
Type Help80.9969.5795.6589.1397.8372.2271.33
Claude-Opus-4.6TRPG - Cold Wind Howling79.8668.13100.0090.1198.3567.5068.56
TRPG - Spring Snow Incident79.5167.9496.1893.1397.3369.0967.36
TRPG - Terror on the Orient Express79.0168.8297.8595.7098.9259.2167.50
No Case Should Remain Unsolved82.2477.7888.89100.00100.0071.4358.33
Type Help82.5371.7482.6191.3097.8383.3369.78
Gemini-3.1-Pro-PreviewTRPG - Cold Wind Howling73.1158.24100.0091.2198.9052.5056.85
TRPG - Spring Snow Incident73.4456.4998.4790.8499.6258.1856.09
TRPG - Terror on the Orient Express71.8054.55100.0093.0599.2053.2553.60
No Case Should Remain Unsolved87.0585.1985.19100.00100.0085.7162.50
Type Help79.7769.5773.9191.3098.9172.2267.52
DeepSeek-V4.1-FlashTRPG - Cold Wind Howling77.6862.64100.0094.51100.0065.0062.77
TRPG - Spring Snow Incident75.7059.54100.0090.0899.2463.6460.12
TRPG - Terror on the Orient Express77.1965.2498.9397.3398.9358.4462.39
No Case Should Remain Unsolved89.7985.19100.0096.30100.0085.7177.09
Type Help81.2271.7486.9693.4898.9172.2267.80
Qwen3-VL-32B-InstructTRPG - Cold Wind Howling69.1052.75100.0092.3194.5150.0052.05
TRPG - Spring Snow Incident63.8443.5196.1888.5593.1345.4545.29
TRPG - Terror on the Orient Express71.1458.2997.3389.3093.3251.9556.87
No Case Should Remain Unsolved72.8666.6781.4892.5996.3064.2937.50
Type Help79.7669.5776.0984.7895.6577.7869.89

Table R1-2. Main results of the replacement models in the in-context configuration.

ModelTRPG (QA Overall)No Case Should Remain Unsolved (QA Overall)No Case Should Remain Unsolved (MFCO)No Case Should Remain Unsolved (MFU)Type Help (QA Overall)Type Help (MFU)
DeepSeek-V4.1-Flash54.1353.8825.0083.3349.7290.00
Gemini-3.1-Pro-Preview29.6149.7475.0090.7420.7069.09
AuthorsOpenReview ↗

Response to Reviewer ranq (R2)

Table R1-3. Six-dimensional QA results of the replacement models in the in-context configuration.

ModelScenarioOverall (%)Acc. (%)Inst. (%)Cit. (%)Read. (%)Comp. (%)Dep. (%)
DeepSeek-V4.1-FlashTRPG - Cold Wind Howling52.6346.1596.7092.3152.7545.0045.04
TRPG - Spring Snow Incident57.7551.1590.8498.4762.2145.4549.76
TRPG - Terror on the Orient Express52.0045.9996.2687.7052.1444.1644.96
No Case Should Remain Unsolved53.8844.4492.5992.5970.3742.8625.00
Type Help49.7236.9689.1386.9658.7044.4437.00
Gemini-3.1-Pro-PreviewTRPG - Cold Wind Howling26.8223.08100.0040.6624.1815.0022.40
TRPG - Spring Snow Incident38.8533.5998.4769.4739.6925.4530.98
TRPG - Terror on the Orient Express23.1719.7997.8634.7617.6516.8818.48
No Case Should Remain Unsolved49.7437.0496.3096.3064.8142.8620.84
Type Help20.7015.2297.8332.6118.4811.1116.04

Table R1-4. Diagnostic results of the replacement models in the in-context configuration.

ModelScenarioFS (%)MS (%)RS (%)
DeepSeek-V4.1-FlashTRPG - Cold Wind Howling47.6245.2550.00
TRPG - Spring Snow Incident53.9253.3254.52
TRPG - Terror on the Orient Express44.8243.0446.61
No Case Should Remain Unsolved56.0256.9655.07
Type Help48.6447.2750.00
Gemini-3.1-Pro-PreviewTRPG - Cold Wind Howling23.8221.9825.67
TRPG - Spring Snow Incident34.9432.7737.11
TRPG - Terror on the Orient Express19.9717.5922.35
No Case Should Remain Unsolved58.4361.7155.15
Type Help20.4619.1321.78
AuthorsOpenReview ↗

Follow up about our response to Reviewer ranq

Dear Reviewer ranq,

Thank you again for your thoughtful comments and constructive suggestions, which have guided our revisions and additional experiments. We really appreciate your careful review and feedback. As the discussion period draws to a close, we remain available to provide any further clarification you may find helpful. If you have any remaining questions or concerns about our response, please feel free to let us know.

We sincerely appreciate the time and care you have devoted to reviewing our work.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear AC and Reviewers,

We sincerely thank you for your careful evaluations and constructive feedback, which have helped us strengthen the experimental validation and improve the clarity of MemGround. We have revised the manuscript and highlighted the changes in blue for your convenience.

In addition to the point-by-point responses below, we summarize the main experiments and clarifications incorporated during the rebuttal phase.


Oracle-Evidence Evaluation (Reviewers ranq and 74ir)

We added oracle-evidence experiments across five models and five scenarios in Appendix E.10, together with complete baseline results for the replacement models. Directly providing annotated supporting evidence improves QA performance, but does not eliminate errors. These results support our finding that both evidence acquisition and subsequent reasoning contribute to long-horizon performance limitations.

Evaluation Reliability and Reproducibility (Reviewer 74ir)

We report three-run QA results with standard deviations, clarify experimental settings in Section 4.2, and expand the five-rater human evaluation and judge-agreement analysis in Appendix E.3. The results show limited variation across QA runs and positive configuration-level correlations between human and automated judgments, supporting the stability of the reported performance patterns.

Metric Definitions and Hint Mechanism (Reviewer 74ir)

We clarified the MFU denominators (Section 4.3, Appendix D.4), hint implementation (Section 4.2, Appendix F.2), Comp. and Dep. calculations (Appendix E.1), retrieved set construction (Appendix E.6.1), and diagnostic classification rules (Appendix E.6.2), and incorporated the no-hint results into Appendix F.2. These clarifications establish that MFU measures overall information coverage, including hint-assisted acquisitions, while the model must still actively select and read a hinted file to acquire its contents.

Native Context Management Baseline (Reviewer R3Ec)

We added a GPT-5.2 native context management and compaction baseline in Appendix E.9. It improves QA Overall across all scenarios relative to the basic GPT-5.2 in-context setup, suggesting that native context management can support more effective long-horizon memory behavior. Remaining limitations in temporal reconstruction and cross-fragment integration highlight challenges beyond retaining information.

Cross-Model Insights (Reviewer 74ir)

We expanded the cross-model analysis in Section 4.1.1 to highlight findings beyond individual rankings. These analyses show that high information coverage does not necessarily yield accurate temporal reconstruction, and that external memory benefits vary across models and scenarios, emphasizing the importance of evidence selection and integration.


We provide detailed responses to each reviewer’s comments and questions below and would be happy to address any further questions or concerns.

Best,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.