- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CompactBench, which measures different models/agents on how well they can compact long contexts critical for long-horizon tasks. The benchmark consists of 200 working-context compaction tasks based on existing agentic tasks from SWEBench (coding), BrowseComp (search), and CompanyBench (workspace). Given a successful agent execution trajectory, the task design introduces some compaction boundary at an intermediate step within the trajectory (at 25%, 50% and 75% of the trajectory length) - the trajectory before the boundary is compacted by the model being evaluated and the same execution agent that produced the full trajectory then continues the task based on the compacted context. The final task result serves as a proxy for the compaction quality. Insights include that different task domains (coding, search, workspace) show different stage-dependent performance trends.
Critical strengths
- The paper evaluates different LLMs on their ability to compact working-context. This capability is critical for long-horizon tasks and the evaluation of this capability via downstream task completion given the compacted context is novel and strongly motivated.
- The analysis is performed for different types of tasks (coding, search, workspace), providing different, nuanced insights for each of them.
Critical weaknesses
My comments primarily concern the experiment design:
-
The selected benchmarks are not ideal to study context compaction & the compaction boundaries seem to be set at arbitrary points (25%, 50%, 75%). The ability of LLMs to compact context is more relevant for their performance on longer horizon tasks. For example, in coding, there are several benchmarks (e.g. SWE-Marathon, Terminal-Bench, LongCLI-Bench) that have much longer horizon tasks, and the insights based on tasks from these benchmarks would be more valuable than on tasks from SWE-Bench. Moreover, compaction becomes relevant once the finite context window of the models are reached, and a more informed boundary selection method is missing.
-
This second point is related to the first. The constraint of only allowing T-b steps after the compaction is not very fair and realistic. The original agent that produced the full trajectory with T steps takes T-b steps after b original steps to complete the task. Compacting the initial b steps and then limiting the agent to the same amount of steps (T-b) as the original agent, which had the entire context of b steps rather than the compacted one up until then, leads to many budget failures hiding potential insights into more interesting/informative failure modes. Ablations on this constraint and a better informed decision on that constraint are missing. It could also work without constraint with the performance being measured by how many post-compaction steps the agent takes to complete the task (more steps could suggest that the compaction was worse, fewer steps could suggest that the compaction is better).
Questions for authors
Questions are related o the weakness comments.
- Have you considered to run the analysis on tasks with very long horizons? The insights for those would be more interesting and relevant.
- Have you run any experiments without or different post-compaction step constraints? If so, what did you find? If not, why not?
4 comments
Response to reviewer y8rh (part 2)
W2 & Q2: Is the budget fair, and what happens without this constraint?
A2: Thank you for this suggestion and for proposing an alternative evaluation design. We match the post-compaction budget to the number of actions that the successful reference execution used after the boundary. Our question is whether the compacted context supports completing the remaining work as efficiently as the original history did, so that extra search or reconstruction cannot compensate for state lost during compaction. The budget is identical for all compactors, so comparisons among them remain fair. Following your suggestion, we added a relaxed-budget experiment.
Relaxed-budget experiment. For each domain, we selected 10 tasks spanning the three boundaries, covering both continuations that had succeeded and continuations that had exhausted their budget in our main experiments. We reran the same Claude Opus 5 compacted context with a budget of actions, i.e., the full length of the reference trajectory, which provides 1.3, 2, and 4 times the matched budget at the 25%, 50%, and 75% boundaries, respectively. Each task was run once.
| Benchmark | Succeeded within | Succeeded after , by | Unsuccessful by |
|---|---|---|---|
| BrowseComp | 1 | 5 | 4 |
| SWE-bench | 0 | 9 | 1 |
| CompanyBench | 3 | 6 | 1 |
| Total | 4 | 20 | 6 |
With actions, 4 of the 30 continuations finished within actions, 20 succeeded only after using more than actions, and 6 remained unsuccessful. Of these 6, four exhausted the full budget , and two terminated before reaching it.
The two settings lead to the same conclusions. This result supports your intuition that many failures under the matched budget are budget failures. The relaxed budget, however, complements rather than contradicts the matched budget: both settings reflect the same compaction failure modes identified in our analysis (RQ4).
- Compaction loss mostly appears as additional actions. Of the 26 continuations that had not finished within actions, 20 went on to complete the task with additional actions, indicating that these continuations were typically still making progress rather than stuck. This matches our failure analysis: in the main experiments, 91.6%, 92.0%, and 97.7% of failed continuations in Search, Coding, and Workspace, respectively, exhausted their budgets; the executor attempted to recover lost information in 95.1% of applicable failed continuations; and omissions were associated with repeating already completed work. The matched budget therefore captures the cost of re-acquiring or reconstructing state that compaction failed to preserve.
- Distorted task states are not repaired by additional actions. RQ4 also shows that state distortions, such as premature commitment, lead the executor to act on an incorrect task representation, and that premature commitment is more frequent in early failures than in budget exhaustion. The relaxed budget shows the same pattern: two of the six unsuccessful continuations terminated before exhausting the budget.
Together, the two settings provide complementary views of the same phenomenon: the matched budget measures whether compaction preserves efficient continuation, while the relaxed budget reveals whether the lost state can still be recovered at a higher cost. We will include the relaxed-budget results in the revision as a complementary analysis.
Follow-up
Dear Reviewer y8rh,
Thank you again for your time and your thoughtful review. As the discussion period closes on November 18 (AoE), we wanted to kindly follow up on our response. In brief, we clarified that CompAct-Bench already covers long trajectories averaging 122.8K tokens in Workspace and that our pipeline applies directly to longer-horizon suites (W1 & Q1), explained that stage-based boundaries reveal domain-specific stage effects that window-triggered boundaries would miss (W1), and confirmed with the relaxed-budget experiment you suggested that both budgets lead to the same conclusions (W2 & Q2). We would be grateful to know whether our response addresses your concerns. We would be happy to clarify anything further before the deadline.
Gentle Follow-up as Discussion Ends
Dear Reviewer y8rh,
As the discussion period ends in about a day, we would greatly appreciate it if you could let us know whether our responses address your concerns. If they do, we would be grateful if you would consider updating your score. We would also be happy to answer any remaining questions before the deadline.
Response to reviewer y8rh (part 1)
Thank you for your careful review and for recognizing that evaluating compaction through downstream task completion is novel and well motivated, and that the analysis across coding, search, and workspace tasks provides nuanced insights. We address each concern below.
A1: Thank you for these suggestions on task horizon and boundary selection. We address the two parts below.
Horizon. We agree that compaction matters most in long-horizon tasks, and CompAct-Bench is not restricted to short trajectories. Workspace trajectories average 58.5 interaction rounds and 122.8K tokens, and Search trajectories average 30.4 rounds, with a long tail of up to 194 rounds and 248.7K tokens. SWE-bench represents an extreme cost-constrained setting: even when the accumulated history would fit in the context window, compacting it to 10% of its length corresponds to compacting proactively to reduce context-processing cost. Its trajectories are also not uniformly short, with individual trajectories reaching 30 rounds or 25.4K tokens, nearly three times the domain average.
Our pipeline requires only runnable tasks, successful reference trajectories, resumable environment states, and automatic evaluators, so it applies directly to longer-horizon suites such as SWE-Marathon, Terminal-Bench, and LongCLI-Bench. The main obstacle is cost: every instance requires three successful source executions, environment replay, and three continuations for each of twelve compactors, and this cost grows rapidly with the horizon. We will discuss this direction in the revision and add it to our future work.
Boundaries. We place boundaries at 25%, 50%, and 75% of execution to compare compaction at the early, middle, and late stages of the same tasks under a common protocol. A boundary triggered by the context window would depend on the window and tokenizer of a particular executor. For a given task, it would also yield only late-stage compaction, which confounds execution stage with absolute context length. The stage-controlled design is what reveals that stage effects differ across domains: accuracy decreases toward later boundaries in Search, increases in Coding, and is non-monotonic in Workspace (RQ2). Combined with the fixed 10% retention target, each boundary emulates a point at which accumulated context must be condensed to a fixed budget, whether because the window is full or because context is compacted proactively to control cost. We will state explicitly that our setting is a controlled, stage-based evaluation rather than a window-triggered deployment simulation.