- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ReConBench to test whether an agent can continue to complete a task after compressing its history, thus evaluating context compression. The method selects a breakpoint from the successful trajectory, resumes the working area, compresses the history into a text memory, hands it over to a new agent for continued execution, and finally checks the task output.
Critical strengths
The evaluation objectives are of practical significance. Compared to summary similarity or information question answering, the ability to continue completing the task after compression more directly reflects whether the memory retains the information needed for the agent's subsequent actions. Furthermore, switching agents to perform the same task is also practically relevant, such as the collaborative work of Claude Code and Codex.
The experimental design and diagnostic information are abundant. The authors compared different compression models, memory formats, budgets, and interruption points, and provided a control group that continued execution without compression. Raw-prefix continuation reached 91.7%, while compressed continuation was approximately 57.3%.
Critical weaknesses
The evaluation has limited coverage of real-world agent handover scenarios. Currently, the source agent and the continuing agent use the same model series and scale, but handover across models and runtime frameworks is not yet covered. For example, a user might switch from Claude Code to Codex mid-task, expecting the latter to continue working based on existing files and compressed memories. Different agents may have different tool interfaces, execution strategies, and information requirements; therefore, effective memories within the same model may not support such handover. Existing results mainly support context compression and recovery within the same model configuration, and validation of memory portability across agents remains insufficient.
Questions for authors
Have you evaluated the baseline of "providing only tasks and workspaces, without providing historical memories"? Furthermore, under the same budget, can repeatedly running the same memory method achieve a result close to the successful union of the four methods?