acceptodds
Under review as a conference paper at ICLR 2027

Coding Agent Memory Post-training: Unlocking the Memory Potential of Pre-trained File Operations for Long-Horizon Tasks via Reinforcement Learning

17 views

Abstract

Language-model agents increasingly tackle long-horizon tasks whose interaction histories exceed the model's active context. Recent work has begun to use reinforcement learning to make memory control part of the policy, often relying on predefined memory tools within domain-specific training environments of relatively short horizons. This setup ties learned memory behavior to environment-specific interfaces that lie outside the base model's pre-training and must be learned from scratch, so even after post-training, agents struggle to use memory in long-horizon tasks. To address these limitations, we introduce Coding Agent Memory Gym (CAMG), a suite of long-horizon agentic-RL environments spanning Shop, Coding, DeepResearch, and AutoResearch. Alongside each environment's native task interface, CAMG provides executable shell access and an episode-persistent workspace, enabling agents to create, revise, search, and reuse files as memory throughout an episode. We also introduce CAMG-RL, which trains a single policy jointly across all four environments with fully asynchronous PPO, learning this file-based memory behavior directly from downstream task reward, and we train CAMG-RL-4B and CAMG-RL-9B from Qwen3.5 models of matching size. On SWE-bench Verified and MLE-bench Lite, CAMG-RL-4B and CAMG-RL-9B are competitive with Qwen3.5-35B-A3B and Qwen3.5-122B-A10B, respectively.

Primary area:
reinforcement learning
Keywords:
Agentic reinforcement learning, Coding agent, Agent memory, Long-horizon agent, Asynchronous reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XG7JOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how language-model agents can learn to retain information across context boundaries through ordinary filesystem operations. Motivated by coding models’ familiarity with file operations, they introduce CAMG, a suite of RL environments spanning shopping, coding, research, and ML-engineering tasks, each equipped with shell access and a persistent workspace. CAMG-RL trains a shared policy using asynchronous PPO with response-level advantages and token-level clipping. The trained models achieve the highest average success rate on CAMG and competitive performance against larger frozen models on external benchmarks.

Critical strengths
  • Carefully controlled evaluation across diverse tasks. CAMG provides four environments with explicit data separation and external contamination checks. Tool-renaming, call-format, and documentation controls strengthen the interface-prior analysis.
  • Strong empirical support and statistical reporting. Memory-content interventions show that saved information contributes to later success. Paired confidence intervals, repeated training runs, and replication of AgeMem’s original recipe strengthen the main comparisons.
Critical weaknesses
  • The gains are not shown to come from memory. There is no RL baseline on the same environments without a memory workspace, and no full-context upper bound. On Coding, CAMG-RL ties with AgeMem, which never uses memory successfully. The average is also largely driven by the synthetic Shop environment.
  • The mechanism relies heavily on environment scaffolding. The environment triggers checkpoint writes, validates them before resetting context, and sometimes penalizes failures, qualifying the “task reward only” framing. The continuation-only variant also trails CompactionRL, suggesting that the advantage depends on additional memory files rather than checkpointing alone.
Questions for authors
  • How do you distinguish difficulty learning AgeMem’s tools from integration issues? Does CAMG-RL’s advantage persist after resolving tool-call parsing failures and aligning context handling?
  • How do you separate the gains from memory learning from improvements in general task competence? Could you compare CAMG-RL against a memory-free RL baseline trained on the same environments with the same training budget, alongside a full-context reference?
3 comments

Thank you for the detailed responses and additional experiments, which partially address my concerns, particularly regarding performance beyond Shop and AgeMem’s use of workspace files on Coding. However, the long-context RL baseline uses automatic compaction and different context settings, leaving the contribution of memory learning insufficiently separated from general task competence. The additional AgeMem analyses also do not establish whether CAMG-RL’s advantage persists after resolving parsing issues and aligning context handling. I therefore appreciate these clarifications but maintain my original score.

Thank the authors for the responses. The additional experiments and explanations are clear. I raise my score.

Thank you for your positive reassessment. We appreciate your constructive feedback and are glad that our additional experiments and clarifications have addressed your main concerns.

Reviewer_Ci49OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The main claim of this paper is that the file read/write capabilities already acquired by foundation models during pretraining can be leveraged for cross-session memory after suitable post-training. To this end, the authors construct a cross-domain RL training environment and adapt PPO to the target task, training a cross-domain, file-based memory policy. Finally, the authors demonstrate the method's effectiveness through comprehensive experiments.

Critical strengths

The idea of eliciting the foundation model's innate capabilities to realize memory is quite valuable, and the experimental results strongly support this claim. In my opinion, this approach taps into the native capabilities of foundation models without requiring additional tools or architectural modifications. In theory, it can generalize to different foundation models and harnesses, enabling a broad range of application scenarios. Nevertheless, the paper does not present experimental results across other models or harnesses, which is a minor limitation.

Critical weaknesses

My major concern lies in the limited strength of the paper’s contributions. The proposed environment and dual-granularity PPO constitute only incremental contributions based on fully investigated methods. Moreover, as one of the claimed contributions, the performance gains from dual-granularity PPO are only partially evaluated in the appendix (see Question 3). In addition, agent memory research over the years has evolved into sophisticated multi-level and multi-dimensional structures. This paper only investigates flat cross-session memory realized via file operations. It remains unclear whether such a mechanism can be extended to hierarchical memory systems and long-term memory across episodes.

Questions for authors
  1. Since the full system prompts under different experimental configurations are not provided in the paper, I cannot determine whether performance discrepancies across models are affected by prompt design. For instance, the paper states that the environment prompts the agent to write memory files when switching context. After switching to a new environment, is there a system prompt instructing the agent to read memories from files? For models pre-trained to use proprietary tools for memory, were they equipped with corresponding system prompts?
  2. The train-free baseline would also follow instructions to write memory entries when receiving environmental prompts. Is there a comparative analysis of memory content to verify that the trained model achieves superior memory distillation and utilization? Is this capability transferable such that the agent can perform better when adopting other memory tools?
  3. I do not see comparison experiments against turn-only PPO in the paper. How can the authors justify that dual-granularity PPO outperforms turn-only PPO on the studied tasks?
  4. Why are these four domains selected for joint training? What performance would be observed if training were conducted on only one of these domains?
  5. Why does the evaluated configuration w/o general files in the ablation study yield a score of 0 on the Shop domain? This result deviates substantially from the trend observed in the other three domains.
  6. Several related works are missing from the citations (e.g., MemGym, SWE-MeM). Filesystem-Based Memory is highly relevant and has closely aligned insights, yet it is not discussed in detail.
Reviewer_a4Y5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The main idea is simple and intuitive. a long running workspace/environment per episode is created mainly by repurposing the current existing benchmarks. A tool as in simple terminal based file use is provided for the agents, but during the long trajectory the conext is wiped out. The idea is that by RL training via only task reward in such a pressure, agents learn to invoke and use memory effectively to deal with the loss of knowledge during the episode. The paper shows this approach leads to improvement mostly within their own task setup but also a limited generalization study is conducted as well.

Critical strengths
  • The main idea is quite interesting and despite simplicity shows potential and effective improvement within the task.
  • The paper is easy to read and written clearly
  • The created environments for long horizon RL training is useful.
Critical weaknesses
  • Despite simplicity, the main intuition is interesting and the provided results are promising, but I am not yet convinced the reported results effectively show the improvements and gains are due to the generalizable learned memory skill while teaching the models using task reward only.

  • The main results provided in Table 2 are promising but not yet convincing since, while wiping the memory mid-training episode is fine, during evaluation such a scenario is arbitrary and synthetic and does not create fair comparisons for approaches that were not trained on such a setting, especially training-free approaches like Mem0. Perhaps specific prompting that warns the agent about such a likely event could have helped make the comparison fairer.

  • Also, the baselines covered here are quite limited. A simple search of the SOTA in this space suggests many other training-free and training-based approaches should be considered and evaluated against.

  • While Table 3 shows promise in terms of generalization to out-of-distribution data, the baselines here are base models and not RL trained on the tasks without such context-losing pressure. A good control here could have involved training on the CAMG tasks on the same 4B and 9B models but without such pressure to make sure any gain reported here is not merely the result of standard RL on task reward and is attributed to the newly learned memory skills.

Questions for authors

see weakness section

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.