- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents MemGym, an evaluation framework that inserts a memory module into long-horizon agents. Four tracks wrap interactive benchmarks, while two new offline QA tracks test retention of debugging observations and research evidence. The authors also train MemRM, a 1.7B scorer that predicts whether compression preserves the next tool action's execution output. Results show task-dependent effects: rolling summaries do not help on SWE-Gym, while context management yields large gains on ProgramBench.
Critical strengths
- Evaluating memory through downstream agent performance is important for practical applications.
- Data contribution. The constructed CodeQA and DR tracks address the recoverability of earlier information from the environment.
- MemRM is a potentially useful contribution for cheaper feedback on compression quality. The released trajectories and construction pipelines may also facilitate follow-up research.
Critical weaknesses
- The framework focuses on within-task working memory. The two constructed tracks are offline QA, while memory-policy training and cross-task memory are not evaluated. This limits the demonstrated utility as a general-purpose memory training environment, as general purpose memory might also concern cross-task memory retention ("continual learning").
- The coding-agent baselines are limited. SWE-Gym tests only rolling summarization against unmanaged history, and the ProgramBench baseline stops at the context limit. Stronger alternatives, such as structured summaries or observation masking, are needed to interpret these results.
- The validity of MemRM as a general memory evaluator is unclear. Although the framework is advised as a gym, the potential to deploy it for RL training is still unclear.
Questions for authors
- Could the authors evaluate the structured summarizer already used on other tracks on SWE-Gym, ideally alongside a simple observation-masking baseline?
- How does MemRM behave under different levels of compression? For example, does it assign sensible scores to the original context (no compression) and an extremely compressed context (almost everything removed)? Do its scores also reflect the quality of realistic compaction methods?
- Could the authors measure how much the agent's behavior varies naturally due to sampling, even without compression? This would provide a useful baseline for MemRM. I am also interested in evaluating agreement over multiple steps rather than just the next tool call. Even with aggressive compression, an agent might still take the correct next action but diverge several steps later.