acceptodds
Under review as a conference paper at ICLR 2027

MemGym: a Long-Horizon Memory Environment for LLM Agents

10 views

Abstract

Long-horizon LLM agents must keep deciding which observations and tool outputs to hold in context. Most memory benchmarks test recall in multi-turn chat. Agent benchmarks where these decisions arise report only end-task success, which mixes memory failures with reasoning and tool-use failures. We present MemGym, an agentic memory benchmark that inserts a memory module between the interaction history and the reasoning model. Its six tracks range from tool-use dialogue and deep-research search to coding and computer use. Each track supports controlled comparisons of memory modules against a track-specific no-memory reference under a fixed environment, scaffold, and reasoner. Four tracks wrap agent benchmarks where forgotten information is often recoverable from the environment. We therefore construct MemGym-CodeQA and MemGym-DR at controlled context lengths with construction-time checks of fact recoverability and answer support. To make SWE-Gym comparisons cheaper, we train MemRM, a 1.7B-parameter scorer that predicts whether the next action after a compression returns the recorded reference output. This local check costs one reasoner step and one classifier call. Full rollouts still measure end-task outcome. We evaluate eight memory strategies with four reasoning models. A rolling summary does not improve SWE-Gym performance. Both summary strategies yield higher mean scores on tool-use dialogue, and a structured summary does so on web tasks. ProgramBench episodes often outgrow the context window. A structured summary raises their mean fraction of hidden tests passed from 9% to 49% under the same step budget. We release MemGym and MemRM with the labeled trajectory corpus and pipeline code at https://anonymous.4open.science/r/MemGym-003C.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Agent Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zpGHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MemGym, an evaluation framework that inserts a memory module into long-horizon agents. Four tracks wrap interactive benchmarks, while two new offline QA tracks test retention of debugging observations and research evidence. The authors also train MemRM, a 1.7B scorer that predicts whether compression preserves the next tool action's execution output. Results show task-dependent effects: rolling summaries do not help on SWE-Gym, while context management yields large gains on ProgramBench.

Critical strengths
  1. Evaluating memory through downstream agent performance is important for practical applications.
  2. Data contribution. The constructed CodeQA and DR tracks address the recoverability of earlier information from the environment.
  3. MemRM is a potentially useful contribution for cheaper feedback on compression quality. The released trajectories and construction pipelines may also facilitate follow-up research.
Critical weaknesses
  1. The framework focuses on within-task working memory. The two constructed tracks are offline QA, while memory-policy training and cross-task memory are not evaluated. This limits the demonstrated utility as a general-purpose memory training environment, as general purpose memory might also concern cross-task memory retention ("continual learning").
  2. The coding-agent baselines are limited. SWE-Gym tests only rolling summarization against unmanaged history, and the ProgramBench baseline stops at the context limit. Stronger alternatives, such as structured summaries or observation masking, are needed to interpret these results.
  3. The validity of MemRM as a general memory evaluator is unclear. Although the framework is advised as a gym, the potential to deploy it for RL training is still unclear.
Questions for authors
  1. Could the authors evaluate the structured summarizer already used on other tracks on SWE-Gym, ideally alongside a simple observation-masking baseline?
  2. How does MemRM behave under different levels of compression? For example, does it assign sensible scores to the original context (no compression) and an extremely compressed context (almost everything removed)? Do its scores also reflect the quality of realistic compaction methods?
  3. Could the authors measure how much the agent's behavior varies naturally due to sampling, even without compression? This would provide a useful baseline for MemRM. I am also interested in evaluating agreement over multiple steps rather than just the next tool call. Even with aggressive compression, an agent might still take the correct next action but diverge several steps later.
Reviewer_rGojOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MemGym, a benchmark framework for evaluating memory mechanisms in long-horizon LLM agents across six tracks covering coding, tool-use dialogue, web interaction, and research tasks. It provides a unified memory-module interface and controlled comparisons with fixed reasoning models and environments. The authors also construct two memory-focused datasets, MemGym-CodeQA and MemGym-DR, and train MEMRM, a 1.7B-parameter model for efficiently evaluating memory compression events. Experiments across eight memory strategies and four reasoning models demonstrate varying benefits of memory management across different tasks

Critical strengths
  1. Diverse evaluation settings. MemGym covers six tracks spanning different agentic environments, providing a broader evaluation of memory mechanisms beyond conventional conversational memory benchmarks.

  2. Controlled evaluation framework. The framework isolates the memory module as an experimental variable while keeping the reasoning model, tools, and environment fixed within each track.

  3. Efficient memory evaluation. MEMRM provides a lightweight proxy for evaluating compression events without requiring complete agent rollouts. The reported results show strong predictive performance on held-out examples, suggesting potential value for efficient memory-system development.

Critical weaknesses
  1. Limited validity of the MEMRM evaluation objective. MEMRM predicts whether the next action produces the same output as a recorded reference action, rather than whether the compressed memory preserves information necessary for eventual task success. Different but valid actions may be penalized, while matching outputs do not guarantee successful downstream execution. Furthermore, the high AUROC is primarily demonstrated on synthetically perturbed compression events, and the paper does not establish whether MEMRM reliably ranks realistic memory strategies.

  2. Insufficient controlled comparisons and ablations. Different tracks evaluate different subsets of memory strategies, limiting systematic cross-domain conclusions. The substantial ProgramBench improvement (9.3% to 48.7%) partly reflects the fact that the no-memory baseline terminates when its context window is exhausted, whereas the structured-summary agent can continue execution.

Questions for authors
  1. MEMRM validity: How well does MEMRM generalize from synthetic memory corruptions to naturally generated summaries and realistic memory failures? Can the authors evaluate whether MEMRM scores correlate with downstream task success and preserve the relative ranking of different memory strategies?

  2. Experimental fairness and ablations: Could the authors compare multiple compaction strategies on ProgramBench under matched context and execution budgets? In particular, how much of the reported improvement comes from enabling continued execution versus preserving more useful historical information? Could the authors also provide more systematic comparisons of the same memory strategies across multiple tracks?

Reviewer_7mFsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies memory evaluation in long-horizon agentic tasks for LLM agents. To fill the gap between non-agentic memory-based benchmarks and agentic benchmarks without explicit memory evaluation, the authors introduce MemGym, which includes a general method for explicitly integrating memory modules into existing agentic benchmarks (wrapped tracks), while also introducing two new datasets with memory management requirements (constructed tracks). To reduce the evaluation cost, they further introduce MemRM, a lightweight scoring model to predict whether the next action after memory compression produces the same tool output as the original trajectory. Experiments are conducted across six tracks with different memory strategies, showing mixed results on wrapped tracks and varying performance across memory methods on constructed tracks. MemRM achieves strong performance on local output agreement prediction and shows a positive correlation with downstream task outcomes.

Critical strengths
  1. The studied problem is important for LLM agent evaluation. Long-horizon memory management is becoming increasingly important as LLM agents handle more complex tasks with growing interaction histories.

  2. The effort to control for the recoverability of historical observations is valuable. Without such controls, agents may obtain the required information directly from the environment instead of relying on memory, leading to biased memory evaluation. The authors attempt to address this issue through explicit recoverability checks when constructing the two new benchmarks.

Critical weaknesses
  1. The choice of evaluated memory methods is limited. All evaluated memory baselines are from before 2026, and none of the existing memory methods are evaluated on the four wrapped agentic benchmarks. This limits the evaluation of the proposed framework.

  2. Unclear role of MemRM. Although it achieves good performance on local output agreement prediction, it is not used to evaluate or compare memory methods in the proposed benchmarks. It is unclear how MemRM actually helps with memory evaluation.

  3. The contribution of the four wrapped tracks is limited. They mainly introduce a memory module into existing agentic benchmarks and still rely on the original task success metrics. It is unclear what additional memory evaluation capabilities they provide compared to directly evaluating memory methods on existing benchmarks.

  4. The effectiveness of the two constructed tracks is not sufficiently validated. Both tracks are offline QA tasks, and the paper does not show whether the performance on these benchmarks reflects memory management capabilities in real interactive agentic tasks.

  5. Lack of in-depth experimental analysis. The paper reports different performance across memory methods and benchmarks, but provides little analysis on why certain methods perform better or worse, with limited ablation studies and failure analysis.

Questions for authors

Please see the weaknesses above for my main questions and concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.