- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Scroll stores agent history losslessly in an append-only event log. The model retrieves from it and computes over it by writing Python in a persistent kernel, and only printed output enters the context. A tiered headline index points to evicted spans. With Qwen3.8-Max it reports 94.8 on LongMemEval-S, 75.5 on BEAM-10M and 93.3 on LOCA-256K. It also includes a backbone-swap table and a 200-probe ablation on BEAM-10M.
Critical strengths
-
The motivation is clear. Compression and external-memory methods decide what to keep before future needs are known, so anything they drop is lost. Scroll keeps the full history and delays selection to query time, when the model writes code to retrieve and compute only what the task needs. This is a principled and useful framing.
-
The lossless-retention ablation supports the central claim. Replacing raw history with rolling summaries lowers the BEAM-10M score by 9.9 points, and by up to 33.8 on knowledge update. The case studies show the cause: details dropped at write time cannot be recovered later.
Critical weaknesses
-
The novel part shows no effect, so the contribution is engineering integration. Scroll is assembled from existing pieces: CodeAct-style code actions, programmatic tool calling, RLM-style REPL access to history, event-sourced logs, and BM25 search. The one new component, the tiered eviction index, has no measurable effect in the authors' own ablation (75.5 vs. 75.0, within the CIs). The rest of the gain comes from known ideas: in the only same-backbone comparison (Table 6), plain CodeAct already reaches 85.3 vs. 93.3. More experiments in a rebuttal cannot add novelty. The submission reads as an industrial prototype rather than a research contribution.
-
The evaluation cannot support the paper's main claims, and fixing it would mean redoing the whole evaluation. The headline claims (+7.5 on BEAM-10M, +44.0 on LOCA) come from Table 2, which compares against published numbers obtained with different backbones, judges and budgets. Table 3 shows that the backbone alone moves LOCA-256K from 93.3 to 22.7. On top of this, the BEAM and LongMemEval prompts inject per-category hints and use a judge from the same model family as the agent. Making these claims valid would require re-running all baselines under one controlled setup and re-running Scroll without hints across the benchmarks. That is a new evaluation and a new paper, not something a rebuttal can deliver. The remaining controlled evidence (a 200-probe ablation on one benchmark and a small appendix table) is at workshop scale.
Overall, this paper is interesting but lower than the standard of ICLR, so I give a clear rejection. I recommend the authors to resubmit to a workshop or add more experiments to meet the standard of ICLR.
Questions for authors
- Are the
[hint]s chosen using the ground-truth question category? Please report results without them. - Please re-run key baselines (e.g., Mem0, RLM, MemGPT) on the same backbone and judge.
- Please report results with a non-Qwen judge.