acceptodds
Under review as a conference paper at ICLR 2027

PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning

Abstract

Long-horizon tasks require sustained perception, reasoning, and exploration, and are a persistent challenge for large language model (LLM) agents. This gap is reflected in their limited performance on continual learning benchmarks such as ARC-AGI-3 and NetHack, especially when evaluated without specialized harnesses. Various agent harnesses have been proposed to close the gap, and each commits to a strategy for handling long sequences of observations, i.e., what information to save from the environment and how to load it into model context, a choice we argue is particularly consequential. Existing methods for context management have typically been designed around a tradeoff, where preserving more information makes relevant details harder to retrieve. Recent progress in coding agents, however, has made new approaches to memory viable. We propose **PRO-LONG**, a minimal context management framework built around *programmatic memory* for LLM agents in long-horizon, exploratory settings. PRO-LONG addresses the tradeoff by keeping a complete, structured interaction log and using programmatic tools to search this history efficiently. On NetHack, we find increasing gains from using PRO-LONG for cross-episodic memory, with scores reaching up to **2.2×** those of a base coding agent. On the full ARC-AGI-3 public game set, PRO-LONG improves over this baseline agent by an average of **18.0** percentage points across frontier models, and matches or exceeds state-of-the-art specialized harnesses (up to **97.4%** best@2) while using **4.2–5.8×** fewer tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-horizon reasoning, agent memory, context management, large language model agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KW5VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how agents can retain and use historical information during extended interactions. It proposes PRO-LONG, a framework that stores actions, observations, and related interaction information in complete external logs, allowing agents to retrieve and analyze their history on demand using tools such as search and Python. The core idea is to use coding agents’ existing programmatic capabilities to preserve details while controlling how much information enters the model’s context. The authors evaluate this design on ARC-AGI-3 and NetHack, reporting performance improvements over a baseline without automatic logging and cost advantages over some specialized harnesses. Ablation studies examine the effects of log length, tool capabilities, and context size.

Critical strengths

The proposed method is simple and practical, and it is evaluated on actual interactive tasks. Automatically preserving complete trajectories avoids losing details through premature compression and allows agents to use code to inspect state changes, test hypotheses, and plan actions. The evaluations on ARC-AGI-3 and NetHack demonstrate its application in these settings. The paper provides useful performance and cost analyses. The authors examine the effects of log length, active context size, tool capabilities, and workspace persistence. The finding that a simple framework can approach the performance of specialized harnesses has practical value.

Critical weaknesses

However, the paper has the following weaknesses:

  1. Limited methodological novelty. Storing information externally and accessing it programmatically on demand have already been explored in works such as Recursive Language Models and Coding Agents are Effective Long-Context Processors. This paper primarily extends that approach to continuously generated interaction trajectories, without introducing a clearly new memory representation, retrieval mechanism, or training method.
  2. The experiments do not isolate the contribution of long-term memory from those of observation access and tool interfaces. The default no-log baseline primarily receives the current board through the prompt, whereas PRO-LONG can directly access per-action logs and animation frames programmatically. Consequently, the improvements may reflect a combination of more complete observations, more accurate state parsing, and access to longer histories.
  3. The evidence for cross-episode learning is insufficient. The NetHack evaluation primarily reports the best score achieved within the first several episodes. This metric can increase with the number of attempts even without learning. The paper lacks a control that preserves identical within-episode logging while clearing only cross-episode history, making it difficult to determine how much of the improvement comes from reusing past experience.
Questions for authors
  1. If the baseline and PRO-LONG receive exactly the same current observations, intermediate states from the preceding action batch, and programmatic access interface, differing only in whether earlier history is retained, how much of the performance advantage remains?
  2. On NetHack, what happens if complete within-episode logs are retained, but cross-episode logs and all other cross-episode information are cleared after each episode?
  3. What new insights and innovations does this paper offer beyond existing approaches to programmatic long-context processing?
4 comments

We thank the reviewer for recognizing the simplicity and practical value of our approach, and for the helpful suggestions for clarifying where the memory gains come from. We address these points and clarify our contribution below.

W2. The experiments do not isolate the contribution of long-term memory from those of observation access and tool interfaces. The default no-log baseline primarily receives the current board through the prompt, whereas PRO-LONG can directly access per-action logs and animation frames programmatically. Consequently, the improvements may reflect a combination of more complete observations, more accurate state parsing, and access to longer histories.

Q1. If the baseline and PRO-LONG receive exactly the same current observations, intermediate states from the preceding action batch, and programmatic access interface, differing only in whether earlier history is retained, how much of the performance advantage remains?

We agree that understanding the impact of programmatic interface and length of history is helpful. We first clarify the no-log baseline variants and their motivations, and then go over our results.

  • Our main no-log baseline (“in-prompt”) follows the reference agents for ARC-AGI-3 and BALROG, which receive observations directly in context [1]. This reflects our motivation to identify simple modifications that improve base coding agents.

  • Our truncated log ablation in Figure 3 varies the amount of history while keeping the observation format, animation frames, and programmatic interface the same. It keeps the last 25 actions and their resulting states, providing more history than the reviewer’s proposed control of keeping only the preceding batch. Across our ARC-AGI-3 setup, batches are capped at 20 actions and average approximately 4-5 actions.

  • To ablate just the programmatic interface, we have added a current board-file baseline, which writes the latest board to a file each turn and overwrites the previous board. The reviewer’s proposed control falls between this baseline and our truncated log in how much history it provides. We agree that testing more history lengths would be helpful, but as each ARC-AGI-3 repetition is approximately $900, we prioritized ablating file access and history length separately within our academic research budget.

Below we extend the best@k comparison from Figure 3 with our added current board-file baseline (n=3 runs), which lets the agent read the board with code but provides no earlier history.

ARC-AGI-3 scorepass@1best@3best@5
PRO-LONG41.23%54.78%60.07%
Last-25 log36.95%48.19%51.94%
No-log (current board file, new)31.68%41.54%–
No-log (base coding agent)24.05%31.72%34.16%

Best@k averages each game's best score across all combinations of k runs. The new baseline has n = 3 runs and the other conditions n = 5. The base coding agent receives the observation directly in its user prompt.

We find that file access explains part of the gain. The board file raises pass@1 from 24.05% to 31.68%, closing less than half of the gap to PRO-LONG (41.23%), while history provides the remaining gains. History older than the last 25 actions adds nearly as much as the last 25 actions do (4.28 against 5.27 pp). Therefore, we find that the gains needs both a board that code can read and the history around it. Neither one alone reaches PRO-LONG.

We emphasize that the truncated log is itself long-context memory. In the truncated log setting, observations from the last 25 actions average about 156,000 tokens, around 60% of Codex's effective context window in our setup. We would expect diminishing returns with longer history, since recent observations are often the most relevant. Even so, the full log improves best@5 from 51.94% to 60.07%. It also does not cost more, and in fact, we found that full-log runs used about 6% fewer billed tokens.

Token usageFull log (PRO-LONG)Last-25 log
Uncached input1.266M1.228M
Cached input45.139M49.971M
Output211.9K213.6K
Billed tokens7.051M7.507M

Average per game, five repetitions across 25 games. Billed tokens follow Appendix A, Eq. (3).

We will add the new baseline to Figure 3 and Section 3.1, clarify how much of the gain comes from file access and how much from history, and include the token table to the appendix.

[1] Reference agents for ARC-AGI-3 (https://github.com/arcprize/ARC-AGI-3-Agents/blob/main/agents/templates/llm_agents.py#L99-L124) and NetHack (via the BALROG benchmark suite) (https://github.com/balrog-ai/BALROG/blob/main/balrog/agents/naive.py#L28-L39) receive observations directly in the model's context.

AuthorsOpenReview ↗

Cross-episodic memory

W3. The evidence for cross-episode learning is insufficient. The NetHack evaluation primarily reports the best score achieved within the first several episodes. This metric can increase with the number of attempts even without learning. The paper lacks a control that preserves identical within-episode logging while clearing only cross-episode history, making it difficult to determine how much of the improvement comes from reusing past experience.

Q2. On NetHack, what happens if complete within-episode logs are retained, but cross-episode logs and all other cross-episode information are cleared after each episode?

We note the question of whether PRO-LONG’s cross-episodic memory provides benefits, beyond repetition effects alone, can be assessed from our data.

In its first episode, PRO-LONG has the complete log of the current episode and no log from earlier episodes. Therefore, drawing five first-episode scores and taking the mean and maximum across them gives an estimate of the baseline with any persistent state cleared between episodes. If sequential episodes behaved like independent attempts, we would expect observed best@5 to be near this estimate. We show both in the table below.

GPT-5.6 + Codex, seed 0Mean of episode 1Observed Mean best@5Estimated best@5 from independent first episodes
No-log3.95%13.58.65%
PRO-LONG4.88%24.514.65%
Kimi K3 + Mini-SWE, averaged over three seeds
No-log3.87%6.26%7.05%
PRO-LONG4.16%12.41%7.87%

Table: Observed vs estimated best@5 for NetHack. To clarify, observed best@5 is each lifetime’s highest score across its five sequential episodes. This is averaged across lifetimes for each seed. The independent attempt estimate averages the highest score in every group of five first episodes. Our additional Kimi seeds are further described in our response to reviewer soPT, W3.

PRO-LONG reaches an observed mean best@5 of 24.5%, against an estimated 14.65% for Codex. Therefore about 10 points of its best score can be attributed to learning from earlier episode experience, not from extra attempts. For Kimi, the gap is about 5 points. No-Log’s corresponding gaps are approximately +5 and −1 points, suggesting that its persistent state, like conversation history or notes, provides less consistent benefits.

We will add this table to Section 3.2, and revise Figure 5 to show the best score on one side and the mean per episode on the other.

While multi-seed ablations that separately clear each form of persistent state are beyond our academic research budget, the above results, and our workspace ablations (like Table 3) suggest that the log is most consequential for PRO-LONG’s performance. We will therefore clarify that PRO-LONG’s benefits may reflect a mix of within-episode and cross-episode memory use in Section 3.2. However, we note that these combined benefits still support our broader thesis of improved long-horizon performance through programmatic memory.

AuthorsOpenReview ↗

Novelty and contribution

W1. Limited methodological novelty. Storing information externally and accessing it programmatically on demand have already been explored in works such as Recursive Language Models and Coding Agents are Effective Long-Context Processors. This paper primarily extends that approach to continuously generated interaction trajectories, without introducing a clearly new memory representation, retrieval mechanism, or training method.

Q3. What new insights and innovations does this paper offer beyond existing approaches to programmatic long-context processing?

We emphasize how PRO-LONG differs from recursive language models (Zhang et al., 2025) and coding agents as long-context processors (Cao et al., 2026) below.

We first note that fixed-input, long context processing tasks are significantly different from long-horizon reasoning tasks. The mentioned works evaluate retrieval and reasoning over fixed inputs using benchmarks such as OOLONG, BrowseComp-Plus, and LongBench-v2. To our knowledge, no prior work does a systematic evaluation on both ARC-AGI-3 and NetHack. These benchmarks matter because they are much harder than retrieval; in retrieval, the answer is already in the input, and the model has to find it. ARC-AGI-3 and NetHack are instead interactive tasks where agents must explore and infer the task dynamics. The agent has to work out what to do from its own play, so it only does well if it continuously learns over long horizons -- in NetHack, we find this can exceed 20,000 game actions

It is therefore unclear how prior insights translate to long-horizon settings, where actions continually generate new observations and the harness must decide what information to keep across many turns, especially because: (1) The scale and horizon of these settings differ significantly. In Cao et al., the shown trajectory for LongBench is 17 search/read operations. To contrast, the median of our ARC runs is around 372 tool calls per game. (2) New challenges, like how memory systems interact with context compaction, become important. Shorter retrieval and QA trajectories are unlikely to hit compaction thresholds. We observe up to ten compactions in individual ARC runs, and vary the compaction threshold (Figure 4) to study how memory supports performance under this repeated context compression.

Our primary insight is how to practically improve base coding agents on long-horizon tasks. Given the cost of these settings, it is especially useful to identify simple changes practitioners can make to their agent harnesses. It is deliberate that PRO-LONG adds no complicated retrieval system, custom memory representation, subagents, or training method. When a system combines several components, it is difficult to understand what drives its performance. Our experiments show that an “append-only” log accessed through code is a useful, low-cost addition to a base coding agent. We compare against specialized harnesses, including RLM-inspired approaches like Arcgentica, and find that PRO-LONG is competitive despite its simplicity. This is desirable, as simple harnesses can translate to significant token savings (~4-5× fewer billed tokens in our evaluations).

We hope this addresses the reviewer's concern about limited novelty, and we will state this motivation more clearly in the revised paper.

Thank you to the authors for their thorough responses. My other concerns are largely addressed, but the important one regarding novelty remains. I am willing to change my opinion regarding novelty during the following discussion phase. For now, I will maintain my original rating.

Reviewer_soPtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how LLM agents can manage context in long-horizon, exploratory tasks. The authors propose PRO-LONG, which keeps a complete log of the interaction in a file and lets a coding agent search it with tools such as grep and Python, instead of summarizing the history or choosing what to store. On the 25 public ARC-AGI-3 games, adding PRO-LONG to a base coding agent improves the score across three frontier models. It comes close to or exceeds specialized harnesses while using 4~5x fewer tokens. On NetHack, where the log is kept across five episodes of the same game, PRO-LONG roughly doubles the base coding agent’s best-of-5 score.

Critical strengths
  • The simplicity of the approach is appreciated. It needs no learned retrieval or memory heuristics, and it works across several model families (GPT, Claude, Kimi) and harnesses (Codex, Claude Code, and Mini-SWE).
  • The cost results are convincing. At matched cost, PRO-LONG scores higher than the specialized harnesses, and it reaches comparable scores with much fewer tokens.
  • Ablations provided on context size, tool access, and log truncation are informative.
Critical weaknesses

Main concerns are:

  1. Gains may not come from long-horizon memory: The truncated log, which keeps only the last 25 actions, already reaches about 37% pass@1 (Figure 3). The full log reaches 41.2% and no-log 24.0%, so most of the gain does not need the long history. The full log helps more at best@5, but even there the truncated log accounts for about two-thirds of the gain over no-log.
  • Also, with read-only tools, PRO-LONG is no better than no-log. The largest jump comes from adding Python.
  • Much of the gain may therefore come from giving the agent the board in a file that code can read. In the no-log baseline, the board appears only in the prompt, so the agent must copy it into a file before analyzing it with code. No-log also spends 30% of its tool calls writing files, versus 4.6% for PRO-LONG (Table 4).
  • The MCP ablation (Figure 7) does not address this, because the tool still returns the board into the context rather than a file. A no-log baseline that writes only the current board to a file each turn is needed to separate these effects.
  1. Claims of matching or beating specialized harnesses are overstated: PRO-LONG scores below the strongest prior harness in two of the three matched comparisons, 41.2% vs. 45.1% for WorldModeler (GPT-5.5), 82.1% vs. 84.4% best@2 for Schema (Fable 5). The cost savings are clear, but the accuracy claims should be toned down.
  2. Weak NetHack evaluation: All NetHack results use a single game seed. Because every episode replays the same game, memory across episodes may exploit the identical starting state rather than reflect general learning. The PRO-LONG prompts also tell the agent to use Python for specific analyses (e.g., tracking explored routes and stairs, BFS), while the no-log prompts do not.

Other concerns:

  • Many ablations are single runs: Apart from the default setting, the context-size results in Figure 4 are single runs, and the paper does not say how many runs Table 2 and Figure 7 use, while run-to-run variation is large.
  • Generality beyond grid games: Both benchmarks have text observations that code can parse directly. It is unclear whether searching a raw log is as useful for tasks with visual or unstructured observations, such as web or GUI agents.
  • Limited novelty: Keeping the full history in a file that a coding agent queries with code is close to existing uses of coding agents and REPLs for long contexts, and the contribution is mainly empirical.
Questions for authors
  • Could the authors add a no-log baseline that writes the current board to a file each turn? This would separate the benefit of history from the benefit of a board that code can read.
  • How does PRO-LONG perform if the log keeps only environment outputs (actions, boards, scores), without the agent’s plans and analyses?
  • For NetHack, could the authors report results on more seeds?
4 comments

We thank the reviewer for recognizing the simplicity and cost efficiency of our approach, and for the helpful suggestions on baselines and evaluation, which we address below.

W1. Gains may not come from long-horizon memory

Q1. Could the authors add a no-log baseline that writes the current board to a file each turn? This would separate the benefit of history from the benefit of a board that code can read.

We appreciate this suggestion as the baseline separates the two effects directly. We have added a no-log baseline that writes the current board to a file before each decision and overwrites the previous board as the reviewer suggested. In this baseline, the agent can read the board with code but sees no earlier history (n = 3 runs).

We note our main no-log baseline follows the reference agents for ARC-AGI-3 and NetHack, which receive observations directly in context [1].

ARC-AGI-3 scorepass@1best@3best@5
PRO-LONG41.23%54.78%60.07%
Last-25 log36.95%48.19%51.94%
No-log (current board file, new)31.68%41.54%–
No-log (base coding agent)24.05%31.72%34.16%

Best@k averages each game's best score across all combinations of k runs. The new baseline has n = 3 runs and the other conditions n = 5. The base coding agent receives the observation directly in its user prompt.

The reviewer is right that file access explains part of the gain. The board file raises pass@1 from 24.05% to 31.68%, which closes less than half of the gap to PRO-LONG (41.23%), while history gives the rest. History older than the last 25 actions adds nearly as much as the last 25 actions do, 4.28 against 5.27 pp. Therefore, the gain needs both parts, a board that code can read and the history around it. Neither one alone reaches PRO-LONG.

We note the truncated log is itself long-context memory. Its last 25 observations average about 156,000 tokens, which is about 60% of Codex's effective context window in our setup. We would expect diminishing returns with longer history, since recent observations are often the most relevant. Even so, the full log improves best@5 from 51.94% to 60.07%. It also costs no more, as full-log runs used about 6% fewer billed tokens.

Token usageFull log (PRO-LONG)Last-25 log
Uncached input1.266M1.228M
Cached input45.139M49.971M
Output211.9K213.6K
Billed tokens7.051M7.507M

Average per game, five repetitions across 25 games. Billed tokens follow Appendix A, Eq. (3).

We will add the new baseline to Figure 3 and Section 3.1, state how much of the gain comes from file access and how much from history, and add the token table to the appendix.

[1] Reference agents for ARC-AGI-3 (https://github.com/arcprize/ARC-AGI-3-Agents/blob/main/agents/templates/llm_agents.py#L99-L124) and NetHack (via the BALROG benchmark suite) (https://github.com/balrog-ai/BALROG/blob/main/balrog/agents/naive.py#L28-L39) receive observations directly in the model's context.

W2. Claims of matching or beating specialized harnesses are overstated: PRO-LONG scores below the strongest prior harness in two of the three matched comparisons, 41.2% vs. 45.1% for WorldModeler (GPT-5.5), 82.1% vs. 84.4% best@2 for Schema (Fable 5). The cost savings are clear, but the accuracy claims should be toned down.

The reviewer is right about the accuracy claims, as PRO-LONG’s average score is below the strongest prior harness on two of three models. We note that those baselines release only one or their best scoring runs, so their variance across multiple runs is unknown, and so is the total number of runs they made to get those best scores. Our GPT-5.5 result is the mean of five runs, with a bootstrap confidence interval of ±3.5 (Table 3). We will soften the wording and replace “matches or exceeds” in the abstract and Section 1 with “competitive with”.

W3. Weak NetHack evaluation: All NetHack results use a single game seed. Because every episode replays the same game, memory across episodes may exploit the identical starting state rather than reflect general learning. The PRO-LONG prompts also tell the agent to use Python for specific analyses (e.g., tracking explored routes and stairs, BFS), while the no-log prompts do not.

Q3. For NetHack, could the authors report results on more seeds?

Yes, we have added results on two more NetHack seeds for Kimi K3, and show the comparison below. We find that PRO-LONG improves average score and best@5 on all three seeds, averaging 1.95x and 2.08x gains, respectively. We find that the second seed follows a similar pattern to the original, where PRO-LONG improves significantly at later attempts, and that the first seed shows strong consistent progress for PRO-LONG, averaging 8.66% vs 3.47% for no-log.

We will incorporate the results into Figure 5 and include individual plots in the appendix.

PRO-LONG meanNo-Log meanProlong best@5No log best@5
Original Seed (Figure 5)4.40% ± 0.72.69% ± 0.211.3%5.1%
New Seed 18.66% ± 1.713.47% ± 1.112.71%8.36%
New Seed 25.98% ± 3.443.49% ± 0.613.26%5.31%

NetHack results for Kimi K3 + Mini-SWE, comparing PRO-LONG and No-Log on the original seed (n=15 lifetimes per condition) and two additional random seeds (n=5 per condition). Best@5 refers to the highest score across a single lifetime’s five attempts.

For defining “lifetimes,” we deliberately fix the seed across episodes. Frontier language models already know the general NetHack strategy from pretraining. NetHack procedurally builds a different dungeon for each seed, with thousands of different plausible enemy encouters, characters, and resources, so most lessons about one map do not carry over to another, and the space of settings is too large to cover within a practical number of episodes. Fixing the seed allows the agent to build on usable insights from previous attempts, analogous to retrying the same game in ARC-AGI-3. Additionally, NetHack is probabilistic, which means encounters and combat outcomes are significantly variable. This means learning how to respond to encounters in a generalizable manner, rather than merely memorizing/replaying action sequences, is an important part of improvement across attempts.

We will clarify our motivation for lifetimes in the paper. We will also qualify in the text that our NetHack prompts for PRO-LONG specify example ways of programmatically accessing a log, while our prompts for no-log specify that helper function programs and notes can be helpful across episodes.

W4. Many ablations are single runs: Apart from the default setting, the context-size results in Figure 4 are single runs, and the paper does not say how many runs Table 2 and Figure 7 use, while run-to-run variation is large.

We have made the number of runs explicit below and will clarify these counts in the main text. We emphasize that all main results use multiple runs. We have also added further repetitions for Figure 7 (n=3 for MCP baseline).

We highlight that, to our knowledge, prior work has not systematically evaluated agents across both ARC-AGI-3 and NetHack at this scale, and all our main results use multiple runs. A single 25 game ARC-AGI-3 sweeps costs approximately $900. Within our academic research budget, we have therefore prioritized repeated evaluation of the main comparisons.

To clarify,

  • Table 2: This is our programmatic tool-access ablation. Each ablated setting uses (n=1), while full tooling uses (n=5), drawn from our main results. Because of this, our conclusions focus on large observed differences, where adding search and Python to Read improves performance by 15.2 pp, compared with a smaller 2.9 point gain from adding Write/Edit.

  • Figure 4: The smaller context settings are single run evaluations. PRO-LONG scores 44.3% at 64K context, compared with 41.2% at the default context size, within the default setting’s reported confidence interval, which we will clarify in the main text.

  • Figure 7: Our average comparisons for the primary no-log and MCP ablation use n=5 and n=3 respectively, while the smaller context settings are single run evaluations.

W5. Generality beyond grid games: Both benchmarks have text observations that code can parse directly. It is unclear whether searching a raw log is as useful for tasks with visual or unstructured observations, such as web or GUI agents.

Following our response to W4, we focus on ARC-AGI-3 and NetHack as two major benchmarks that emphasize long-horizon reasoning and discovery of environment dynamics, with trajectories ranging from hundreds to over 10,000 actions in our experiments. We agree that our results do not establish generality to purely visual or unstructured observations.

However, programmatically accessible observations are not unique to grid games. Web agents often have access to the document object model (DOM), a structured representation of a webpage’s content and elements. Similarly, benchmarks like OSWorld and AndroidWorld support accessibility trees that describe interface elements alongside screenshots. Such representations could support programmatic access to interaction history. We believe extending our evaluation to these visually grounded environments is an interesting direction for future work.

W6. Limited novelty: Keeping the full history in a file that a coding agent queries with code is close to existing uses of coding agents and REPLs for long contexts, and the contribution is mainly empirical.

We appreciate the reviewer raising this concern, and we make three main points to show how PRO-LONG differs from recursive language models (Zhang et al., 2025) and coding agents as long-context processors (Cao et al., 2026).

Existing work largely focuses on fixed-input, long context processing tasks. The mentioned works evaluate retrieval and reasoning over fixed inputs using benchmarks such as OOLONG, BrowseComp-Plus, and LongBench-v2. To our knowledge, no prior work does a systematic evaluation on both ARC-AGI-3 and NetHack. These benchmarks matter because they are much harder than retrieval; in retrieval, the answer is already in the input, and the model has to find it. ARC-AGI-3 and NetHack are instead interactive tasks where agents must explore and infer the task dynamics. The agent has to work out what to do from its own play, so it only does well if it continuously learns over long horizons -- in NetHack, we find this can exceed 20,000 game actions

It is therefore unclear how prior insights translate to long-horizon settings, where actions continually generate new observations and the harness must decide what information to keep across many turns, especially because:

  • The scale and horizon of these settings differ significantly. In Cao et al., the shown trajectory for LongBench is 17 search/read operations. To contrast, the median of our ARC runs is around 372 tool calls per game.

  • New challenges, like how memory systems interact with context compaction, become important. Shorter retrieval and QA trajectories are unlikely to hit compaction thresholds. We observe up to ten compactions in individual ARC runs, and vary the compaction threshold (Figure 4) to study how memory supports performance under this repeated context compression.

How can we practically improve base coding agents on long-horizon tasks? Given the cost of these settings, it is especially useful to identify simple changes practitioners can make to their agent harnesses. It is deliberate that PRO-LONG adds no complicated retrieval system, custom memory representation, subagents, or training method. When a system combines several components, it is difficult to understand what drives its performance. Our experiments show that an “append-only” log accessed through code is a useful, low-cost addition to a base coding agent.

We compare against specialized harnesses, including RLM-inspired approaches like Arcgentica, and find that PRO-LONG is competitive despite its simplicity. This is desirable, as simple harnesses can translate to significant token savings (~4-5× fewer billed tokens in our evaluations). We think this finding matters to the community also as it sets a simple yet strong baseline that more complex memory systems have to beat. We hope this addresses the reviewer's concern about limited novelty, and we will state this motivation more clearly in the revised paper.

Q2. How does PRO-LONG perform if the log keeps only environment outputs (actions, boards, scores), without the agent’s plans and analyses?

Thanks for this question. Each log entry contains the action header, the agent’s summarized plan, the action, and the resulting board. We have not run an environment-only ablation, given the cost of ARC evaluation and our prioritization of the controls above.

Our qualitative analysis gives a partial answer to this. In Section 2.2., we focused on examples of the agent leveraging environment outputs, e.g., an agent greps the log for score changes to identify consequential actions, and another writes a regress.py script that replays logged actions to check whether its game model predicts the recorded boards.

Interestingly, we also find explicit use of prior reasoning. In ft09, a grid-coloring puzzle, PRO-LONG retrieves text analysis from ~360 actions earlier to check which interpretations of the color clues had failed before choosing a new mask to test. We will add this qualitative evidence and identify the environment-only ablation as future work in the limitations.

Reviewer_p9pUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PRO-LONG keeps a structured log of an agent's actions and observations. The agent can search this log with file tools and Python instead of keeping everything in its active context. The main ARC-AGI-3 experiments cover all 25 public games with a 500-action budget across three models. The NetHack experiments use five sequential episodes, with additional ablations for tools, context length, and workspace persistence.

Critical strengths
  1. The ARC gains are clear and consistent. PRO-LONG improves pass@1 over no-log across three models. This is a meaningful improvement for a simple modification.

  2. The method is easy to understand and audit. The same logging idea works in Codex, Claude Code, and Mini-SWE. The paper provides prompts, game versions, and token-cost details, which makes the comparison more transparent.

Critical weaknesses
  1. The NetHack result does not prove cross-episode learning. Best@k is a cumulative maximum, so it can rise just because more episodes are tried. A control that clears the log between episodes is missing. The fixed-seed setup and unresolved “Appendix ??” reference also make this evidence hard to interpret.

  2. Some external comparisons are not fully matched. WorldModeler uses vision, and Schema uses different tools and selection rules. Some cost comparisons count retained runs rather than all attempts. The ablations also have limited uncertainty reporting, and adding Python may help general reasoning as well as log retrieval.

Questions for authors
  1. Can you clear the log between NetHack episodes and report per-episode results on several held-out seeds?

  2. Can you provide run counts and uncertainty for the ablations, with a factorial control that separates log access from Python access?

  3. Can you compare end-to-end costs using all attempts for both systems and clarify whether workspace state was reset identically across episodes?

3 comments

We appreciate the reviewer for recognizing PRO-LONG’s consistent gains, simplicity, and transparency, as well as the suggestions for strengthening our evaluation. We address the suggestions and questions below.

W1. The NetHack result does not prove cross-episode learning. Best@k is a cumulative maximum, so it can rise just because more episodes are tried. A control that clears the log between episodes is missing. The fixed-seed setup and unresolved “Appendix ??” reference also make this evidence hard to interpret.

Q1. Can you clear the log between NetHack episodes and report per-episode results on several held-out seeds?

We note the question of whether PRO-LONG’s cross-episodic memory provides benefits, beyond repetition effects alone, can be assessed from our data.

In its first episode, PRO-LONG has the complete log of the current episode and no log from earlier episodes. Therefore, drawing five first-episode scores and taking the mean and maximum across them gives an estimate of the baseline with any persistent state cleared between episodes. If sequential episodes behaved like independent attempts, we would expect observed best@5 to be near this estimate. We show both in the table below.

GPT-5.6 + Codex, seed 0Mean of episode 1Observed Mean best@5Estimated best@5 from independent first episodes
No-log3.95%13.58.65%
PRO-LONG4.88%24.514.65%
Kimi K3 + Mini-SWE, averaged over three seeds
No-log3.87%6.26%7.05%
PRO-LONG4.16%12.41%7.87%

Table: Observed vs estimated best@5 for NetHack. To clarify, observed best@5 is each lifetime’s highest score across its five sequential episodes. This is averaged across lifetimes for each seed. The independent attempt estimate averages the highest score in every group of five first episodes. Our additional Kimi seeds are further described in our response to reviewer soPT, W3.

PRO-LONG reaches an observed mean best@5 of 24.5%, against an estimated 14.65% for Codex. Therefore about 10 points of its best score can be attributed to learning from earlier episode experience, not from extra attempts. For Kimi, the gap is about 5 points. No-Log’s corresponding gaps are approximately +5 and −1 points, suggesting that its persistent state, like conversation history or notes, provides less consistent benefits.

We will add this table to Section 3.2, and revise Figure 5 to show the best score on one side and the mean per episode on the other. While multi-seed ablations that separately clear each form of persistent state are beyond our academic research budget, the above results, and our workspace ablations (like Table 3) suggest that the log is most consequential for PRO-LONG’s performance. We will therefore clarify that PRO-LONG’s benefits may reflect a mix of within-episode and cross-episode memory use in Section 3.2. However, we note that these combined benefits still support our broader thesis of improved long-horizon performance through programmatic memory.

Fixed seed clarification: Across episodes, we fix the seed on purpose. Language models already know the general NetHack strategy from pretraining. NetHack also builds a different dungeon for each seed (with thousands of possible layouts/encounters/characters), so most lessons about one map do not carry over to another, and the space of settings is too large to cover within a practical number of episodes. Fixing the seed allows the agent to build on usable insights from previous attempts, analogous to retrying the same game in ARC-AGI-3. Additionally, NetHack is probabilistic, which means encounters and combat outcomes can be variable across episodes. This means learning how to respond to encounters in a generalizable manner, rather than just memorizing action sequences, is important for improving across attempts. We will clarify our choice in the paper and fix the broken Appendix reference in Section 2.1.

W2(c) The ablations also have limited uncertainty reporting, and adding Python may help general reasoning as well as log retrieval.

Q2. Can you provide run counts and uncertainty for the ablations, with a factorial control that separates log access from Python access?

We would like to note that experiments on the full suite of ARC-AGI-3 benchmark are extremely expensive – a single 25-game ARC-AGI-3 sweep costs ~$900. This prohibits running extensive ablations on an academic budget. Our main comparison is to the no-log baseline, which runs the exact same setup as PRO-LONG, and that we report confidence intervals for. We believe the costs emphasizes the importance of PRO-LONG’s contribution for practitioners, which is that simpler harnesses with significant token saving compared to specialized ones can still be very competitive.

The table below gives the number of runs behind each result.

ResultRuns per setting
Main results, GPT-5.5 (Figures 2 and 3)5
Main results, Claude Code (Figures 2 and 3)2
Table 2, tool ablation1 per ablated setting, 5 for full tools
Figure 4, non-default context sizes1 each, 5 for the default
Figure 7, no-log and MCP5 and 3

Single run settings have no interval of their own, so we compare them with the interval of the default setting (Table 3). The 64K setting falls inside it.

For the factorial control, the paper's settings already cover three of the four cells.

pass@1With PythonWithout Python
Log41.23% (PRO-LONG, 5 runs)23.1% (read-only, 1 run)
No log24.05% (5 runs)– (not run)

We emphasize the log helps only when the agent can read it with code. With Python, the log raises pass@1 from 24.05% to 41.23%. Without Python, the agent with the log does no better than no-log. This matches our thesis that the gain needs both programmatic access and a memory log. The read-only setting also drops search and file editing. We ran it once, because without being able to use code, the agent heavily relies on expensive/large read calls, and a single run costs about \3,000vs.3,000 vs. \\900 for PRO-LONG.

We have also added a no-log baseline that writes the current board to a file before each decision, described in our reply to Reviewer soPt. It has Python but no history, and it reaches 31.68% pass@1, so even when code can read the board, the history adds about 10 points. We will state n and the interval in every ablation caption, and add this table to Section 3.3.

AuthorsOpenReview ↗

Cost Comparisons

W2(a) Some external comparisons are not fully matched. WorldModeler uses vision, and Schema uses different tools and selection rules.

We clarify that Figure 2 is intended to emphasize PRO-LONG’s performance relative to specialized ARC-AGI-3 harnesses, specifically those with the strongest previously reported results for each frontier model. As the reviewer points out, these harnesses incorporate additional components or modifications intended to improve performance, e.g., WorldModeler uses both vision and text, Schema replaces Claude Code’s default tools with custom tools, and Arcgentica uses subagents.

This comparison supports our argument for simplicity, as PRO-LONG is a minimal addition to a base coding agent (using neither vision or subagents), while remaining competitive and cheaper relative to these specialized harnesses. We will clarify that Figure 2 is meant to be system-level comparisons with prior work rather than fully matched ablations. We note our no-log comparisons and other variants are for ablating effects of specific components.

W2(b) Some cost comparisons count retained runs rather than all attempts.

Q3. Can you compare end-to-end costs using all attempts for both systems and clarify whether workspace state was reset identically across episodes?

Thanks for raising this. We read "both systems" as PRO-LONG and the specialized harness in each cost comparison, which we also address as part of W2(b) and make explicit in the table below. We note that our PRO-LONG versus No-log cost comparisons appear in Figure 2, with the token cost accounting described in Appendix A.

Cost. Our Figure 2 reports the cost comparisons for specialized harnesses. To address W2(b), our goal was to evaluate costs relative to the best prior released harness for a given model. WorldModeler releases one run, so we can compare our pass@1 (mean of 5) scores and cost to it. Schema, the best prior for Claude Code, only releases their best of two runs (i.e., retained run). For a principled comparison, we compare our best of two runs to it as well, which we go further in detail to in Footnote 3. To help clarify this, we will add the full cost tables to Appendix A (summarized below) to accompany our Figure 2 plots and add a cost score table for NetHack.

ComparisonPRO-LONG runs countedBaseline runs countedPRO-LONG billed tokens
GPT-5.5 pass@1 vs WorldModeler1 (mean of 5)its released run5.8× fewer
GPT-5.5 best@5 vs WorldModelerall 5its released run1.2× fewer
Fable 5 best@2 vs Schema best@2our retained best of 2 runits retained run (best of 2)4.2x fewer
Fable 5 best@2 vs Schema best@2bothits retained run (best of 2)2.0× fewer

Cost accounting for comparisons with specialized harnesses. The other harnesses release one run per game, so for them we can count only that run. For PRO-LONG we can count every attempt, and the table above does this.

We note that PRO-LONG costs less in every row, even when all of its attempts are counted.

Workspace. The workspace is handled identically in both conditions. It is kept across the five episodes of a lifetime, so any notes or scripts that the agent writes carry over in PRO-LONG and in no-log alike. The log is the only extra memory that PRO-LONG carries across episodes. We will state this in Section 2.1.

Comments on the paper

1 comment

We briefly summarize the main concerns and our responses:

  • Novelty [soPT, KW5V]: We have clarified the differences between the prior work highlighted by the reviewers, which focus on processing fixed long contexts (e.g., retrieval and Q&A), and our work, which studies simple ways to improve long-horizon reasoning on tasks requiring thousands of model actions.

  • Memory versus observation access [soPt, KW5V]: We added the requested baseline [soPt] that makes the current board accessible through a file. PRO-LONG remains 9.6 pp ahead. We noted our base coding agent baseline is based off both benchmarks’ reference agents, and that our existing truncated log baseline further ablates history benefits, addressing [KW5V]’s additional control.

  • Cross-episode learning [p9pU, soPt, KW5V]: We added results for two NetHack seeds, finding gains on all three. We showed through data analysis, comparing repeated episodes against independent attempts, that PRO-LONG’s benefits are not just from having more attempts.

  • Ablations and costs [p9pU, soPt]: We have clarified run counts across ablations and our cost comparisons for both specialized harnesses and baselines.

We sincerely appreciate the thoughtful feedback from reviewers. We thank the reviewers for recognizing PRO-LONG as “a meaningful improvement for a simple modification” [p9pU], that “the cost results are convincing” [soPt], and that “the proposed method is simple and practical” [KW5V].

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.