- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how agents can retain and use historical information during extended interactions. It proposes PRO-LONG, a framework that stores actions, observations, and related interaction information in complete external logs, allowing agents to retrieve and analyze their history on demand using tools such as search and Python. The core idea is to use coding agents’ existing programmatic capabilities to preserve details while controlling how much information enters the model’s context. The authors evaluate this design on ARC-AGI-3 and NetHack, reporting performance improvements over a baseline without automatic logging and cost advantages over some specialized harnesses. Ablation studies examine the effects of log length, tool capabilities, and context size.
Critical strengths
The proposed method is simple and practical, and it is evaluated on actual interactive tasks. Automatically preserving complete trajectories avoids losing details through premature compression and allows agents to use code to inspect state changes, test hypotheses, and plan actions. The evaluations on ARC-AGI-3 and NetHack demonstrate its application in these settings. The paper provides useful performance and cost analyses. The authors examine the effects of log length, active context size, tool capabilities, and workspace persistence. The finding that a simple framework can approach the performance of specialized harnesses has practical value.
Critical weaknesses
However, the paper has the following weaknesses:
- Limited methodological novelty. Storing information externally and accessing it programmatically on demand have already been explored in works such as Recursive Language Models and Coding Agents are Effective Long-Context Processors. This paper primarily extends that approach to continuously generated interaction trajectories, without introducing a clearly new memory representation, retrieval mechanism, or training method.
- The experiments do not isolate the contribution of long-term memory from those of observation access and tool interfaces. The default no-log baseline primarily receives the current board through the prompt, whereas PRO-LONG can directly access per-action logs and animation frames programmatically. Consequently, the improvements may reflect a combination of more complete observations, more accurate state parsing, and access to longer histories.
- The evidence for cross-episode learning is insufficient. The NetHack evaluation primarily reports the best score achieved within the first several episodes. This metric can increase with the number of attempts even without learning. The paper lacks a control that preserves identical within-episode logging while clearing only cross-episode history, making it difficult to determine how much of the improvement comes from reusing past experience.
Questions for authors
- If the baseline and PRO-LONG receive exactly the same current observations, intermediate states from the preceding action batch, and programmatic access interface, differing only in whether earlier history is retained, how much of the performance advantage remains?
- On NetHack, what happens if complete within-episode logs are retained, but cross-episode logs and all other cross-episode information are cleared after each episode?
- What new insights and innovations does this paper offer beyond existing approaches to programmatic long-context processing?
4 comments
Cross-episodic memory
W3. The evidence for cross-episode learning is insufficient. The NetHack evaluation primarily reports the best score achieved within the first several episodes. This metric can increase with the number of attempts even without learning. The paper lacks a control that preserves identical within-episode logging while clearing only cross-episode history, making it difficult to determine how much of the improvement comes from reusing past experience.
Q2. On NetHack, what happens if complete within-episode logs are retained, but cross-episode logs and all other cross-episode information are cleared after each episode?
We note the question of whether PRO-LONG’s cross-episodic memory provides benefits, beyond repetition effects alone, can be assessed from our data.
In its first episode, PRO-LONG has the complete log of the current episode and no log from earlier episodes. Therefore, drawing five first-episode scores and taking the mean and maximum across them gives an estimate of the baseline with any persistent state cleared between episodes. If sequential episodes behaved like independent attempts, we would expect observed best@5 to be near this estimate. We show both in the table below.
| GPT-5.6 + Codex, seed 0 | Mean of episode 1 | Observed Mean best@5 | Estimated best@5 from independent first episodes |
|---|---|---|---|
| No-log | 3.95% | 13.5 | 8.65% |
| PRO-LONG | 4.88% | 24.5 | 14.65% |
| Kimi K3 + Mini-SWE, averaged over three seeds | |||
| No-log | 3.87% | 6.26% | 7.05% |
| PRO-LONG | 4.16% | 12.41% | 7.87% |
Table: Observed vs estimated best@5 for NetHack. To clarify, observed best@5 is each lifetime’s highest score across its five sequential episodes. This is averaged across lifetimes for each seed. The independent attempt estimate averages the highest score in every group of five first episodes. Our additional Kimi seeds are further described in our response to reviewer soPT, W3.
PRO-LONG reaches an observed mean best@5 of 24.5%, against an estimated 14.65% for Codex. Therefore about 10 points of its best score can be attributed to learning from earlier episode experience, not from extra attempts. For Kimi, the gap is about 5 points. No-Log’s corresponding gaps are approximately +5 and −1 points, suggesting that its persistent state, like conversation history or notes, provides less consistent benefits.
We will add this table to Section 3.2, and revise Figure 5 to show the best score on one side and the mean per episode on the other.
While multi-seed ablations that separately clear each form of persistent state are beyond our academic research budget, the above results, and our workspace ablations (like Table 3) suggest that the log is most consequential for PRO-LONG’s performance. We will therefore clarify that PRO-LONG’s benefits may reflect a mix of within-episode and cross-episode memory use in Section 3.2. However, we note that these combined benefits still support our broader thesis of improved long-horizon performance through programmatic memory.
Novelty and contribution
W1. Limited methodological novelty. Storing information externally and accessing it programmatically on demand have already been explored in works such as Recursive Language Models and Coding Agents are Effective Long-Context Processors. This paper primarily extends that approach to continuously generated interaction trajectories, without introducing a clearly new memory representation, retrieval mechanism, or training method.
Q3. What new insights and innovations does this paper offer beyond existing approaches to programmatic long-context processing?
We emphasize how PRO-LONG differs from recursive language models (Zhang et al., 2025) and coding agents as long-context processors (Cao et al., 2026) below.
We first note that fixed-input, long context processing tasks are significantly different from long-horizon reasoning tasks. The mentioned works evaluate retrieval and reasoning over fixed inputs using benchmarks such as OOLONG, BrowseComp-Plus, and LongBench-v2. To our knowledge, no prior work does a systematic evaluation on both ARC-AGI-3 and NetHack. These benchmarks matter because they are much harder than retrieval; in retrieval, the answer is already in the input, and the model has to find it. ARC-AGI-3 and NetHack are instead interactive tasks where agents must explore and infer the task dynamics. The agent has to work out what to do from its own play, so it only does well if it continuously learns over long horizons -- in NetHack, we find this can exceed 20,000 game actions
It is therefore unclear how prior insights translate to long-horizon settings, where actions continually generate new observations and the harness must decide what information to keep across many turns, especially because: (1) The scale and horizon of these settings differ significantly. In Cao et al., the shown trajectory for LongBench is 17 search/read operations. To contrast, the median of our ARC runs is around 372 tool calls per game. (2) New challenges, like how memory systems interact with context compaction, become important. Shorter retrieval and QA trajectories are unlikely to hit compaction thresholds. We observe up to ten compactions in individual ARC runs, and vary the compaction threshold (Figure 4) to study how memory supports performance under this repeated context compression.
Our primary insight is how to practically improve base coding agents on long-horizon tasks. Given the cost of these settings, it is especially useful to identify simple changes practitioners can make to their agent harnesses. It is deliberate that PRO-LONG adds no complicated retrieval system, custom memory representation, subagents, or training method. When a system combines several components, it is difficult to understand what drives its performance. Our experiments show that an “append-only” log accessed through code is a useful, low-cost addition to a base coding agent. We compare against specialized harnesses, including RLM-inspired approaches like Arcgentica, and find that PRO-LONG is competitive despite its simplicity. This is desirable, as simple harnesses can translate to significant token savings (~4-5× fewer billed tokens in our evaluations).
We hope this addresses the reviewer's concern about limited novelty, and we will state this motivation more clearly in the revised paper.
Thank you to the authors for their thorough responses. My other concerns are largely addressed, but the important one regarding novelty remains. I am willing to change my opinion regarding novelty during the following discussion phase. For now, I will maintain my original rating.
We thank the reviewer for recognizing the simplicity and practical value of our approach, and for the helpful suggestions for clarifying where the memory gains come from. We address these points and clarify our contribution below.
We agree that understanding the impact of programmatic interface and length of history is helpful. We first clarify the no-log baseline variants and their motivations, and then go over our results.
Our main no-log baseline (“in-prompt”) follows the reference agents for ARC-AGI-3 and BALROG, which receive observations directly in context [1]. This reflects our motivation to identify simple modifications that improve base coding agents.
Our truncated log ablation in Figure 3 varies the amount of history while keeping the observation format, animation frames, and programmatic interface the same. It keeps the last 25 actions and their resulting states, providing more history than the reviewer’s proposed control of keeping only the preceding batch. Across our ARC-AGI-3 setup, batches are capped at 20 actions and average approximately 4-5 actions.
To ablate just the programmatic interface, we have added a current board-file baseline, which writes the latest board to a file each turn and overwrites the previous board. The reviewer’s proposed control falls between this baseline and our truncated log in how much history it provides. We agree that testing more history lengths would be helpful, but as each ARC-AGI-3 repetition is approximately $900, we prioritized ablating file access and history length separately within our academic research budget.
Below we extend the best@k comparison from Figure 3 with our added current board-file baseline (n=3 runs), which lets the agent read the board with code but provides no earlier history.
Best@k averages each game's best score across all combinations of k runs. The new baseline has n = 3 runs and the other conditions n = 5. The base coding agent receives the observation directly in its user prompt.
We find that file access explains part of the gain. The board file raises pass@1 from 24.05% to 31.68%, closing less than half of the gap to PRO-LONG (41.23%), while history provides the remaining gains. History older than the last 25 actions adds nearly as much as the last 25 actions do (4.28 against 5.27 pp). Therefore, we find that the gains needs both a board that code can read and the history around it. Neither one alone reaches PRO-LONG.
We emphasize that the truncated log is itself long-context memory. In the truncated log setting, observations from the last 25 actions average about 156,000 tokens, around 60% of Codex's effective context window in our setup. We would expect diminishing returns with longer history, since recent observations are often the most relevant. Even so, the full log improves best@5 from 51.94% to 60.07%. It also does not cost more, and in fact, we found that full-log runs used about 6% fewer billed tokens.
Average per game, five repetitions across 25 games. Billed tokens follow Appendix A, Eq. (3).
We will add the new baseline to Figure 3 and Section 3.1, clarify how much of the gain comes from file access and how much from history, and include the token table to the appendix.
[1] Reference agents for ARC-AGI-3 (https://github.com/arcprize/ARC-AGI-3-Agents/blob/main/agents/templates/llm_agents.py#L99-L124) and NetHack (via the BALROG benchmark suite) (https://github.com/balrog-ai/BALROG/blob/main/balrog/agents/naive.py#L28-L39) receive observations directly in the model's context.