acceptodds
Under review as a conference paper at ICLR 2027

Scroll: Programmatic Context Management for Long-Horizon Agents

Abstract

Long-horizon agents for complex software engineering and deep research accumulate actions, observations, and intermediate state far beyond a model's context window. Context management must therefore efficiently construct compact, task-relevant views of this growing history for each model call, even though future information needs are unknown. Existing methods typically either bound context growth through structured but lossy compaction and extraction, or externalize the full history as raw, unstructured context for model-written programs to navigate. We present , a lossless and programmable context manager for long-running agent sessions. An append-only event log retains history losslessly under stable addresses and timestamps, while a tiered eviction index maintains in-context address-based anchors to off-context session history. Model-written programs access history through structured search and recovery operations, compute over it in a sandboxed persistent Python kernel, and expose only selected results. With Qwen3.8-Max, Scroll scores **94.8** on LongMemEval, **75.5** on BEAM, and **93.3** on LOCA, exceeding the best published scores by **7.5** points on BEAM and **44.0** points on LOCA. Ablations on BEAM show that lossless retention, the persistent Python kernel, and the structured access layer make complementary contributions to accurate and token-efficient context construction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, context management, long-horizon agents, agent memory, code generation, context engineering, tool use

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Xo1gOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Scroll stores agent history losslessly in an append-only event log. The model retrieves from it and computes over it by writing Python in a persistent kernel, and only printed output enters the context. A tiered headline index points to evicted spans. With Qwen3.8-Max it reports 94.8 on LongMemEval-S, 75.5 on BEAM-10M and 93.3 on LOCA-256K. It also includes a backbone-swap table and a 200-probe ablation on BEAM-10M.

Critical strengths
  1. The motivation is clear. Compression and external-memory methods decide what to keep before future needs are known, so anything they drop is lost. Scroll keeps the full history and delays selection to query time, when the model writes code to retrieve and compute only what the task needs. This is a principled and useful framing.

  2. The lossless-retention ablation supports the central claim. Replacing raw history with rolling summaries lowers the BEAM-10M score by 9.9 points, and by up to 33.8 on knowledge update. The case studies show the cause: details dropped at write time cannot be recovered later.

Critical weaknesses
  1. The novel part shows no effect, so the contribution is engineering integration. Scroll is assembled from existing pieces: CodeAct-style code actions, programmatic tool calling, RLM-style REPL access to history, event-sourced logs, and BM25 search. The one new component, the tiered eviction index, has no measurable effect in the authors' own ablation (75.5 vs. 75.0, within the CIs). The rest of the gain comes from known ideas: in the only same-backbone comparison (Table 6), plain CodeAct already reaches 85.3 vs. 93.3. More experiments in a rebuttal cannot add novelty. The submission reads as an industrial prototype rather than a research contribution.

  2. The evaluation cannot support the paper's main claims, and fixing it would mean redoing the whole evaluation. The headline claims (+7.5 on BEAM-10M, +44.0 on LOCA) come from Table 2, which compares against published numbers obtained with different backbones, judges and budgets. Table 3 shows that the backbone alone moves LOCA-256K from 93.3 to 22.7. On top of this, the BEAM and LongMemEval prompts inject per-category hints and use a judge from the same model family as the agent. Making these claims valid would require re-running all baselines under one controlled setup and re-running Scroll without hints across the benchmarks. That is a new evaluation and a new paper, not something a rebuttal can deliver. The remaining controlled evidence (a 200-probe ablation on one benchmark and a small appendix table) is at workshop scale.

Overall, this paper is interesting but lower than the standard of ICLR, so I give a clear rejection. I recommend the authors to resubmit to a workshop or add more experiments to meet the standard of ICLR.

Questions for authors
  1. Are the [hint]s chosen using the ground-truth question category? Please report results without them.
  2. Please re-run key baselines (e.g., Mem0, RLM, MemGPT) on the same backbone and judge.
  3. Please report results with a non-Qwen judge.
Reviewer_uiDuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies context management for agents whose interaction histories exceed the model’s context window. The authors propose Scroll, which retains complete history in an append-only event log, exposes structured search and recovery operations, and lets the model write Python programs to retrieve and compute over history. A persistent kernel retains intermediate objects, while a tiered eviction index provides navigation anchors to off-context records. Scroll reports strong results on LongMemEval, BEAM, and LOCA, with ablations examining lossless retention, executable computation, and the eviction index.

Critical strengths
  • The system addresses a practical problem. Keeping original records recoverable avoids committing to a lossy summary before the future information need is known.
  • The integration of historical retrieval, computation, and selective output is clear and useful. It allows the agent to operate on large results without repeatedly putting all of them into the model context.
Critical weaknesses
  • The novelty over the closest programmatic-memory systems is not sufficiently established. PRO-LONG already proposes retaining complete structured interaction logs and using coding agents to search them programmatically. RLM also treats long context as an external object that the model inspects through code. LCM preserves verbatim history with hierarchical navigation and lossless recovery. Scroll adds a particular combination of structured search, persistent objects, and a tiered index, but there is no matched comparison showing what these additions provide over the closest existing designs. PRO-LONG is especially relevant and is missing from the related work. The contribution currently appears mainly to be a systems integration and its evaluation.
  • The memory-benchmark prompts use category information that may advantage the evaluated system. Appendix C supplies different hints for different question categories. For example, the BEAM hints explicitly state that a value changed or that the conversation contains conflicting statements. LongMemEval abstention questions receive a dedicated instruction suggesting that the question may be unanswerable. The selection of this instruction depends on the answerability category. It is unclear whether the published baselines receive the same information. What happens if all questions use the same generic prompt without category or abstention metadata? A no-hint evaluation, or identical hints for all compared methods, is needed to establish the gains under a common protocol.

Minors:

  • The statement in Section 2.1 that reduced representations replace history and make omitted details unrecoverable is too broad. MemGPT and LCM retain original records even when using summaries or compressed working views.
  • Please provide the actual working-context budget, eviction threshold, tier width, protected-tail size, and output budget used in the experiments.
  • Table 3 shows that several backbones can use Scroll, but does not include their corresponding no-Scroll baselines. It therefore does not directly establish the claim that stronger models benefit more from the method.
Questions for authors

See weaknesses

Reviewer_KUKmOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Scroll, which stores the complete interaction history in an external event log. It enables the model to retrieve, recover, and compute over historical information through Python programs, while exposing only selected results to the context. A hierarchical index is further used to navigate historical information that has been evicted from the active context.

Critical strengths

The paper addresses information loss caused by compressing long agent histories. It separately manages raw records, persistent computational states, and the model’s active context, enabling efficient evidence retrieval and long-term task execution. The experiments cover both long-term memory retrieval and interactive tool use, with histories of up to 10 million tokens and evaluations across multiple backbone models. The results demonstrate the value of preserving raw evidence, programmatic computation, and selective output. The category-wise analysis further reveals where different mechanisms are most effective.

Critical weaknesses

Table 2 compares systems using different backbone models, retrieval budgets, and judges, which the authors also acknowledge as a system-level comparison. Therefore, the gain on BEAM and LOCA cannot be directly attributed to Scroll’s context management mechanism. Appendix C.2 provides category-specific guidance, such as looking for the latest value in knowledge-update questions, identifying conflicts in contradiction-resolution questions, and considering that abstention questions may have no answer. These hints provide additional task information beyond data-format instructions. It is unclear whether other baselines receive the same guidance, and no ablation without these hints is reported. The paper combines existing mechanisms such as event logs, full-text search, and CodeAct-style program execution. While this integration has practical value, the novel contribution beyond these existing techniques needs to be more clearly demonstrated. The paper does not provide sufficient details on the eviction threshold, hierarchical index parameters, or how navigation titles are generated during history import. The appendix states that importing requires no additional LLM calls, while the titles appear to be model-generated. This process needs further clarification.

Questions for authors

Could the authors compare Scroll with existing memory systems and a simple baseline of “persistent Python + raw history + full-text search” under the same backbone, prompts, judge model, and inference budget? Are the task-specific hints in the appendix selected based on the ground-truth category labels? Do other baselines use the same information? Please report results without these hints, especially without revealing in advance which questions are abstention cases. Is the 0.5-point improvement from the hierarchical eviction index statistically significant? Could the authors add an ablation that retains Python execution and selective output but removes persistent variables across calls? How are the titles and hierarchical index constructed during history import, and are all associated costs included in the reported results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.