acceptodds
Under review as a conference paper at ICLR 2027

ReConBench: Evaluating Context Compression through Executable Agent Resumption

Abstract

During a long task, an agent accumulates a history of decisions, tool calls, and observations. Compressing this history reduces the context passed to a successor, but may also remove information needed to finish the task. We introduce ReConBench, a benchmark that evaluates context compression through task completion after an interruption. Starting from a verified successful execution, we select a checkpoint, restore its workspace, compress the preceding interaction history into a memory, and ask a fresh agent to complete the task. A task-specific verifier evaluates the final workspace. The benchmark pool contains 2,694 successful trajectories organized into four workflow families. Our main study evaluates 32,400 continuation runs across interruption stages, compressor scales, memory methods, and budgets within three source/continuation-agent strata. Among verifier-valid runs, 57.3% complete the task, with a mean normalized score of 0.834. Success rises from 49.4% at an early checkpoint to 67.8% at a late checkpoint, and from 54.2% with a 2B compressor to 59.6% with a 9B compressor. Memory methods differ by only a few percentage points, and larger allowances produce neither proportional increases in memory length nor consistently higher success. These findings show why compression should be evaluated through both the task it preserves and the work required after handoff.

Primary area:
datasets and benchmarks
Keywords:
Context Compression, Executable Agent Resumption, LLM Agents, Task Completion Benchmark, State Handoff, Long-horizon Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_c4JwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReConBench to test whether an agent can continue to complete a task after compressing its history, thus evaluating context compression. The method selects a breakpoint from the successful trajectory, resumes the working area, compresses the history into a text memory, hands it over to a new agent for continued execution, and finally checks the task output.

Critical strengths

The evaluation objectives are of practical significance. Compared to summary similarity or information question answering, the ability to continue completing the task after compression more directly reflects whether the memory retains the information needed for the agent's subsequent actions. Furthermore, switching agents to perform the same task is also practically relevant, such as the collaborative work of Claude Code and Codex.

The experimental design and diagnostic information are abundant. The authors compared different compression models, memory formats, budgets, and interruption points, and provided a control group that continued execution without compression. Raw-prefix continuation reached 91.7%, while compressed continuation was approximately 57.3%.

Critical weaknesses

The evaluation has limited coverage of real-world agent handover scenarios. Currently, the source agent and the continuing agent use the same model series and scale, but handover across models and runtime frameworks is not yet covered. For example, a user might switch from Claude Code to Codex mid-task, expecting the latter to continue working based on existing files and compressed memories. Different agents may have different tool interfaces, execution strategies, and information requirements; therefore, effective memories within the same model may not support such handover. Existing results mainly support context compression and recovery within the same model configuration, and validation of memory portability across agents remains insufficient.

Questions for authors

Have you evaluated the baseline of "providing only tasks and workspaces, without providing historical memories"? Furthermore, under the same budget, can repeatedly running the same memory method achieve a result close to the successful union of the four methods?

Reviewer_F8HiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReConBench, a benchmark for evaluating context compression through agent task completion after a mid-task interruption. A fresh agent receives the original task, the restored workspace, and a compressed version of the preceding interaction history, then attempts to finish the task under a task-specific verifier. The benchmark contains 2,694 verified successful source trajectories. The main study evaluates 32,400 continuation runs derived from 300 trajectories, comparing interruption stages, compressor sizes, memory formats, and memory budgets. Compressed handoffs achieve 57.3% success, compared with 91.7% for raw-prefix continuation. Later checkpoints and larger compressors yield higher success, while memory format and nominal budget show smaller aggregate differences.

Critical strengths

I find the evaluation setup useful because it tests whether a compressed memory actually helps an agent finish its work. A summary can look reasonable while omitting a failed approach, an unresolved requirement, or the reason behind an earlier decision. Resuming execution makes these omissions consequential and measurable. Preserving the workspace also captures an important part of practical handoffs: some progress is already stored in files, while the interaction history contains information needed to interpret and build on that progress.

The experimental design includes several controls that make the results more informative. Comparing memory constructions at the same source checkpoint holds the task and starting workspace fixed, and the raw-prefix continuation control helps distinguish the difficulty of replacing an agent from the additional loss associated with compression. I also appreciate the reporting of realized memory lengths, downstream execution costs, replay mismatches, and uncertainty clustered by source trajectory and task. These analyses expose limitations that a single success rate would hide and make the benchmark more useful for subsequent work.

Critical weaknesses

The reliability of the judge-based evaluation is not sufficiently established in the paper. Appendix A.5 describes a combination of executable checks and a 9B semantic judge, but it remains unclear how often judge-assessed criteria determine the final pass/fail outcome. The paper also does not report agreement with human assessments on a representative sample. This leaves uncertainty about whether the evaluation is reliable enough to interpret the relatively small performance differences between memory methods. Clarifying the contribution of semantic judgments and validating a sample of them would strengthen confidence in the results.

The evaluation covers a relatively limited range of history lengths. Source trajectories contain at most 29 assistant turns and 28 tool calls, and the histories supplied for compression average approximately 4,240 tokens. This leaves uncertainty about whether the observed small differences between memory formats and nominal budgets extend to substantially longer histories, where retaining earlier decisions and constraints may be more demanding. Experiments on longer histories would help establish the scope of these findings.

Questions for authors

Same as the weakness section.

Reviewer_aZAFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ReConBench evaluates how well agent context compression works by checking whether a fresh agent can finish the task during handoff, by interupting verified successful trajectories. The main study covers a large 32k runs across different source trajectories, interruption stages, compressor sizes, memory methods, and budgets. Experiments conducted using rather small Qwen3.5 2B/4B/9B models. The paper finds that overall success with compressed memory had a significant gap compared to when the agent is given the full uncompressed history. Success rises with later checkpoints and larger compressors. The paper also finds memory method and budget make little difference.

Critical strengths

The paper presented a useful way to evaluate compression. Judging compression by whether a fresh agent can finish the task, instead of by how similar the summary is to a reference, measures what actually matters for agent handoffs.

Critical weaknesses
  • Verifier is uncalibrated. It claimed to be using an unspecified 9B judge. The paper provides no human agreement study, alternative-judge analysis, repeated-judgment variance, or breakdown of deterministic versus semantic criteria.
  • Only one model family is tested. All roles, source agent, compressor, continuation agent and judge use Qwen3.5 at 2B–9B. It is unknown whether the conclusions hold for other open families or frontier models: that larger compressors help, and that method and budget barely matter.
  • Writing issues here and there - Figure 2 text overlap. Token counts in the text don't match Table 8 (e.g., 97,036 vs 97,057). The paper also lacks reproduceability. Specifically the paper does not say which environment, tools or task suite produced the 2,694 source trajectories, how they were collected, or how the 151 tasks were chosen. No info about the judge rubrics were shown either.
Questions for authors
  • What environment, tools and task suite produced the 2,694 trajectories, and how were the trajectories and the 151 tasks selected? Can you share the tasks, harness, compression prompts and verifiers through an anonymous repository during the discussion period?
  • Do the findings hold for other families and frontier models, and when the compressor and the continuation agent come from different families (the paper calls the memory "portable" in Sec. 3.6 but never tests this)? Do they also hold for histories long enough to require compression?
  • Which model is the "9B" judge, why it was chosen, what rubrics does it score, and how well does it agree with humans?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.