- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies context and memory management for long-horizon LLM agents. ScrubAgent integrates task actions with operations for protecting selected context spans, compressing recent interactions, storing discoveries as retrievable keyframes, and recalling relevant evidence. ScrubGRPO trains this policy with outcome rewards and process penalties targeting context pressure, unused keyframes, and information loss. The paper evaluates the approach on a 150-question BrowseComp-Plus split and 500 SWE-bench Verified instances, and reports component ablations, memory-retrieval analyses, context-budget sweeps, and computational costs. Its intended contribution is to combine selective preservation of detailed context with directly retrievable task evidence under a bounded working context.
Critical strengths
-
A practically relevant problem and a concrete, reusable interface. Long-horizon agents must retain decisive evidence while controlling the cost of accumulated tool observations. The proposed combination of protected context spans and retrievable keyframes provides an explicit interface for addressing this tension. Its use in both information-seeking and software-repair tasks suggests potential applicability across different tool environments, although this does not establish zero-shot cross-domain generalization.
-
Useful diagnostic analyses beyond aggregate task scores. The manuscript includes component ablations, retrieval variants, context-budget sweeps, and cost accounting. Subject to clarification of the evaluation protocol and reported numbers, these analyses can help explain when memory preservation and retrieval are useful and what costs they incur. In particular, the distinction between active-context compression and end-to-end latency is useful for assessing practical benefits.
Critical weaknesses
-
Limited demonstrated novelty relative to closely related work. Several central ideas have close precedents. AgeMem integrates storage, retrieval, summarization, and discarding into the agent's policy and trains unified memory management with step-wise GRPO. AgentFold jointly generates context-folding directives and tool actions at each step, preserving useful details through multi-scale condensation. Context Folding/FoldGRPO already learns active context management through end-to-end RL with process rewards. Cognitive Scaffold converts context into structured event snapshots and proactively retrieves precise evidence. These methods differ in their memory representations, training procedures, and task settings, so I do not consider them identical to ScrubAgent. Nevertheless, the manuscript does not sufficiently establish why its particular combination of focus windows, task-local keyframes, and process penalties constitutes a substantial methodological advance. The paper compares against Context Folding and adapted memory methods, but lacks direct comparisons with important close alternatives such as AgeMem and AgentFold. Component-removal ablations establish usefulness within the proposed system, rather than novelty over existing combinations. Matched comparisons against an existing folding agent augmented with the same keyframe store would be especially informative.
-
Limited evidence on model-size scaling and a missing matched control. Increasing the backbone from Qwen2.5-7B-Instruct to 14B improves BrowseComp-Plus from 54.8% to 56.4%, but SWE-bench Verified only from 53.6% to 53.8%. The manuscript itself reports that the latter difference is not statistically significant (p = 0.34). However, it does not report corresponding 14B ReAct/standard-GRPO baselines, making it impossible to determine whether the small gain is intrinsic to this task and training setting or is associated with the proposed memory policy. This is particularly relevant because increasing the 7B working context from 32K to 64K yields larger reported gains of 3.9 and 2.2 percentage points, respectively. A matched model-size-by-method comparison, together with a memory/context-capacity analysis, is needed to establish whether the method preserves the benefits of a stronger backbone. This concern limits the evidence for parameter scaling; it does not by itself refute the separate claim of supporting longer interaction horizons.
-
Unresolved inconsistencies in the reported evaluation and abstract claims undermine confidence in the empirical evidence. Appendices H/I.4/K specify three independent seeds, 150 BrowseComp-Plus tasks, one greedy episode per task, and binary correctness, with scores averaged over seeds. Under this protocol, the mean must be 100k/450 percent for an integer k. However, the reported 54.8% is not attainable after rounding to one decimal: 246 successful episodes yield 54.7%, whereas 247 yield 54.9%. The 54.3% FoldGRPO baseline has the same incompatibility. These observations do not identify whether the error lies in implementation, aggregation, reporting, or the protocol description, but they prevent verification of the headline result and comparisons; the associated confidence intervals and significance tests also require reconciliation with the underlying data. Separately, the supplied OpenReview abstract reports 63.3% / 59.2% without specifying model size, whereas the manuscript abstract reports 54.8% / 53.6% at 7B and 56.4% / 53.8% at 14B. The OpenReview abstract explicitly claims effective generalization to trajectories longer than those seen in training, while the manuscript abstract only states that trajectory-length and retrieval analyses are reported. The core method remains consistent and neither abstract states a formal theoretical guarantee, so this should not be characterized as a change in algorithm or a conflicting theorem. Nevertheless, the authors need to identify the authoritative result version, reconcile the evaluation and aggregation protocol, and clarify the scope and evidence for the generalization claim. Until then, the reliability and scope of the empirical contribution remain uncertain.
-
Insufficient grounding of the human-cognition motivation. The abstract, introduction, and Figure 1 repeatedly motivate the design through claims that movie viewers skim routine content, preserve important scenes at higher resolution, and later recall bookmarked scenes directly. The manuscript does not cite cognitive-science evidence for these specific claims or clearly distinguish established findings from an engineering analogy. Research by Baldassano et al. and Ben-Yakov and Henson on event segmentation and episodic memory supports structured encoding of continuous experience, but does not directly establish the proposed combination of adaptive resolution and explicitly addressable keyframes. The component ablations demonstrate utility for the agent, rather than correspondence with human memory mechanisms. The authors should either substantiate the mapping between cognitive findings and design choices or frame the movie example explicitly as an analogy and ground the technical motivation in measurable evidence-retention and retrieval failures. I view this as a supporting concern about the argument and positioning, rather than a standalone reason to reject an otherwise strong engineering contribution.
Questions for authors
-
Clarification of evaluation aggregation. How can the 54.8% ScrubAgent score and 54.3% FoldGRPO score arise under the stated protocol of three seeds, 150 tasks, and one binary-scored episode per task? Please provide all 150 task IDs and the 3-by-150 binary outcome matrices for the compared methods, per-seed success counts and denominators, and the exact aggregation code. If additional runs, weighting, exclusions, non-binary scores or a different estimator were used, please state this explicitly and correct the protocol description. Please also provide the bootstrap inputs and explain how seed and task variability were handled, then recompute affected means, deltas, confidence intervals and p-values as necessary.
-
Clarification of abstract versions and claim scope. Which model, checkpoint, evaluation split and protocol produced the OpenReview abstract's 63.3% / 59.2%, and how do those results relate to the manuscript's 7B and 14B scores? Do the two abstracts describe the same implementation and experiment version? Does the paper retain the OpenReview abstract's explicit claim of effective generalization beyond training trajectory lengths, and which controlled experiment supports it? The supplied manuscript groups tasks by each method's own executed length; please distinguish this conditional analysis from a comparison on a common fixed task set or a controlled horizon-extrapolation experiment. This is a request to identify the intended claim and its existing evidence, not an assumption that the method has changed.
-
Clarification of the evaluation setup and unusually strong base-agent scores. Table 1 reports 38.0% and 48.8% on SWE-bench Verified for short- and long-context Qwen2.5-7B-Instruct ReAct without the listed RL stage. Please identify the exact checkpoint and any prior training, and release the prompts, environment configuration, predictions, and complete evaluation reports for all 500 instances. Please explain the handling of timeouts, failed patch applications and test failures, and how hidden evaluation artifacts are isolated from the agent. For BrowseComp-Plus, please provide the 680/150 split IDs, split provenance, corpus configuration and scorer, and indicate which reference-model rows were rerun under the identical setup versus imported from other sources. Different setups may explain the absolute scores; the current description is insufficient to verify this explanation.
-
Evidence needed to assess scaling. Can the authors provide 14B ReAct/standard-GRPO results under the same data and evaluation budgets, and establish whether expanding the working context or keyframe capacity changes the 7B-to-14B gain? The matched baseline is central to interpreting the current scaling result; the capacity sweep would help diagnose its cause.
-
Evidence needed to assess the additional contribution. What benefit remains over AgeMem or AgentFold, or over a folding agent equipped with the same keyframe storage and retrieval module? Please distinguish any existing evidence from proposed additional experiments. Matching the maximum 32K context alone does not match actual inference cost: Table 2 reports 210.5K prefill tokens for ScrubAgent versus 78.6K for FoldGRPO. A comparison at comparable actual cost would materially affect the assessment of the additional mechanism.