- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ReScopeBench to evaluate agents' ability to acquire, reuse, and revise experience in stateful environments while maintaining its validity. The benchmark contains 18 simulated environments, each with a sequence of 16 tasks, for a total of 288 tasks. Tasks are organized around shared structural and rule experience, and agents must discover relevant information within limited budgets of environmental resources. Each task stream undergoes a local change before task 9, without an announcement to the agent. The change invalidates some experience while leaving other dependencies intact, and subsequent tasks provide opportunities for revision and reuse.
Evaluation runs two task streams from the same initial state: one retains experience, while the other clears it before each task. Scores based on actual delivered outcomes measure the benefits of experience, the impact of changes, and subsequent recovery. The paper compares seven model deployment configurations. The results show that experience acquired through interaction can improve performance on later tasks, but environmental changes reduce these gains, and recovery varies across configurations. Analyses of tool use and limited content audits also show that fewer interactions do not necessarily indicate more effective experience reuse, and that retaining correct experience does not mean it is actually used during execution.
Critical strengths
-
The paper makes the continued applicability of experience its main evaluation target. Exploration and verification consume environmental resources that are also needed to complete tasks, requiring agents to balance direct reuse against rechecking. Local changes require them to revise the parts that have become invalid while preserving unaffected experience. These design choices focus on maintaining usable experience during sustained operation, extending beyond retrieval, restatement, or the continued accumulation of historical information. I consider this the paper's strongest motivation.
-
The authors make several useful distinctions in task construction and result interpretation. A successful reference execution does not establish that an agent can discover a solution through publicly available observations. Scoring should also be based on actual delivered outcomes and allow different valid completion paths. Accordingly, the paper describes discoverability checks, validation of recovery after changes, and outcome-based scoring. It also distinguishes a reduction in the benefits of experience after a change from experience becoming harmful relative to the condition without it. Likewise, retaining correct experience and using that experience during execution are separate issues. These distinctions help avoid overly simplistic interpretations of the role of experience based only on declining performance or memory contents.
Critical weaknesses
-
The contribution beyond existing work is not sufficiently established. EvoArena already covers changes in external workflows, dependencies, interfaces, permissions, and other operating conditions. The need to update experience after an environmental change is therefore not, by itself, a sufficient distinction. Combining resource budgets, unannounced local changes, and selective revision could add evaluation value, but the paper lacks concrete task materials showing how these requirements arise in practice. The authors describe requirements for budget calibration, discoverability checks, and validation of recovery paths, but do not provide a complete account connecting actual tasks, observable feedback, invalidated and preserved dependencies, resource consumption, and scores. I therefore cannot readily verify whether repeated exhaustive exploration is genuinely unaffordable, whether local investigation and repair are feasible, or whether later tasks actually test reuse of the revised experience. Such materials would provide direct evidence that the additional evaluation requirements are implemented; their role goes beyond illustrating an example for readers. The current results do not yet show which capability differences the benchmark can identify that existing evaluations have difficulty distinguishing.
-
The experience management mechanisms and control conditions are insufficiently specified, and the validation is limited, making it difficult to draw clear conclusions about experience maintenance. The condition with retained experience includes history, memory, and skills, and the behavioral analysis also mentions background consolidation. However, the paper does not clearly explain how these are generated, retrieved, and updated, or how context overflow is handled. The condition without retained experience still preserves environment-managed observation and capability records. Their contents, whether they can be queried, and how they differ from the cross-task information that is cleared also need to be specified. The authors acknowledge that the business states of the two task streams evolve separately, so the comparison measures the longitudinal benefit of retaining experience rather than the effect of intervening on memory alone at an identical intermediate state. I accept this evaluation objective, but it makes a clear definition of the experience system and its information retention boundaries particularly important for interpreting the comparison. In addition, each condition is run only once per environment, and the content audit covers only two environments and relies on a single pass of LLM-assisted annotation. These analyses offer some local observations, but they do not establish whether the experience gains and recovery trends are stable, nor do they connect conflict detection, experience revision, and subsequent reuse into a complete process. Researchers seeking to use this benchmark to compare experience maintenance methods still lack sufficiently detailed experimental specifications and validation evidence.
Questions for authors
-
Compared with dynamic environment evaluations such as EvoArena, what additional capabilities are required by the combination of resource budgets, unannounced local changes, and selective revision? How do these requirements affect agents' decisions, and why are the resulting differences difficult to capture with existing evaluations? Please explain the contribution in these terms, rather than only listing differences in benchmark settings.
-
Could the authors provide a representative task sequence from an actual benchmark environment spanning both sides of a change, connecting the public tasks, tool feedback, shared experience dependencies, invalidated and preserved components, resource budgets, and scores? What are the respective resource costs of informed execution, local diagnosis and repair, and repeated exhaustive exploration? How were these constraints checked during benchmark construction, and are the corresponding validation results available? This information would directly affect my assessment of whether the tasks realize the intended tradeoffs.
-
What experience management procedure is used in each of the seven model deployment configurations? Which components generate interaction history, memory, and skills, and when are these retrieved and updated? Does background consolidation use the same model as task execution, and how is context overflow handled? Are these mechanisms consistent across configurations? What environment-managed observation and capability records remain in the condition without retained experience, can the agent query them, and how do they differ from the history and caches that are cleared? These details are needed to interpret and reproduce the experiments.
-
Could the recovery analysis and content audit use existing trajectories to demonstrate a complete instance of experience maintenance: detecting that old experience is no longer valid, revising the affected parts, reusing the revision in a later task, and continuing to use unaffected experience? This would help distinguish maintenance and reuse from rediscovering information in each task. Are repeated-run results already available to assess the stability of the main experience gains and recovery trends? If not, limited reruns on a small set of representative environments would address this concern more directly than adding further models.