acceptodds
Under review as a conference paper at ICLR 2027

CompAct-Bench: A Benchmark for Working-Context Compaction in Long-Horizon Agent Tasks

Abstract

As language-model agents are applied to increasingly extended tasks, their working context continuously accumulates interaction history, tool observations, intermediate decisions, and task progress. Context compaction offers a practical way to reduce this growing history while allowing execution to continue, yet the compaction ability of modern LLMs remains insufficiently understood. We introduce CompAct-Bench, a benchmark for evaluating working-context compaction through its downstream effect on agent task completion. Starting from successful agent trajectories, we insert compaction boundaries at different stages of execution, compress the preceding working context with the model under evaluation, and let the same executor continue the original task. To enable fair comparison across compactors, we evaluate them under controlled retention ratios using a lightweight iterative ratio-adjustment procedure. CompAct-Bench covers information-seeking, software-engineering, and workspace-oriented tasks through BrowseComp, SWE-bench, and an internal CompanyBench evaluation. Our experiments show that working-context compaction remains highly lossy even for frontier models: Claude- and GPT-family models perform strongest overall, yet substantial degradation in downstream task accuracy persists under aggressive compaction. Further analysis reveals that failures are often caused not only by omitted information, but also by distorted task state, such as turning unresolved issues into apparent conclusions or forgetting previously established results, which can lead to incorrect decisions or redundant execution. These findings highlight working-context compaction as a distinct and still challenging ability for LLMs operating in extended agent tasks.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Working Context, LLM, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_y8rhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CompactBench, which measures different models/agents on how well they can compact long contexts critical for long-horizon tasks. The benchmark consists of 200 working-context compaction tasks based on existing agentic tasks from SWEBench (coding), BrowseComp (search), and CompanyBench (workspace). Given a successful agent execution trajectory, the task design introduces some compaction boundary at an intermediate step within the trajectory (at 25%, 50% and 75% of the trajectory length) - the trajectory before the boundary is compacted by the model being evaluated and the same execution agent that produced the full trajectory then continues the task based on the compacted context. The final task result serves as a proxy for the compaction quality. Insights include that different task domains (coding, search, workspace) show different stage-dependent performance trends.

Critical strengths
  1. The paper evaluates different LLMs on their ability to compact working-context. This capability is critical for long-horizon tasks and the evaluation of this capability via downstream task completion given the compacted context is novel and strongly motivated.
  2. The analysis is performed for different types of tasks (coding, search, workspace), providing different, nuanced insights for each of them.
Critical weaknesses

My comments primarily concern the experiment design:

  1. The selected benchmarks are not ideal to study context compaction & the compaction boundaries seem to be set at arbitrary points (25%, 50%, 75%). The ability of LLMs to compact context is more relevant for their performance on longer horizon tasks. For example, in coding, there are several benchmarks (e.g. SWE-Marathon, Terminal-Bench, LongCLI-Bench) that have much longer horizon tasks, and the insights based on tasks from these benchmarks would be more valuable than on tasks from SWE-Bench. Moreover, compaction becomes relevant once the finite context window of the models are reached, and a more informed boundary selection method is missing.

  2. This second point is related to the first. The constraint of only allowing T-b steps after the compaction is not very fair and realistic. The original agent that produced the full trajectory with T steps takes T-b steps after b original steps to complete the task. Compacting the initial b steps and then limiting the agent to the same amount of steps (T-b) as the original agent, which had the entire context of b steps rather than the compacted one up until then, leads to many budget failures hiding potential insights into more interesting/informative failure modes. Ablations on this constraint and a better informed decision on that constraint are missing. It could also work without constraint with the performance being measured by how many post-compaction steps the agent takes to complete the task (more steps could suggest that the compaction was worse, fewer steps could suggest that the compaction is better).

Questions for authors

Questions are related o the weakness comments.

  1. Have you considered to run the analysis on tasks with very long horizons? The insights for those would be more interesting and relevant.
  2. Have you run any experiments without or different post-compaction step constraints? If so, what did you find? If not, why not?
4 comments
AuthorsOpenReview ↗

Response to reviewer y8rh (part 1)

Thank you for your careful review and for recognizing that evaluating compaction through downstream task completion is novel and well motivated, and that the analysis across coding, search, and workspace tasks provides nuanced insights. We address each concern below.

W1 & Q1: Are the source benchmarks long-horizon enough, and how were the compaction boundaries chosen?

A1: Thank you for these suggestions on task horizon and boundary selection. We address the two parts below.

Horizon. We agree that compaction matters most in long-horizon tasks, and CompAct-Bench is not restricted to short trajectories. Workspace trajectories average 58.5 interaction rounds and 122.8K tokens, and Search trajectories average 30.4 rounds, with a long tail of up to 194 rounds and 248.7K tokens. SWE-bench represents an extreme cost-constrained setting: even when the accumulated history would fit in the context window, compacting it to 10% of its length corresponds to compacting proactively to reduce context-processing cost. Its trajectories are also not uniformly short, with individual trajectories reaching 30 rounds or 25.4K tokens, nearly three times the domain average.

Our pipeline requires only runnable tasks, successful reference trajectories, resumable environment states, and automatic evaluators, so it applies directly to longer-horizon suites such as SWE-Marathon, Terminal-Bench, and LongCLI-Bench. The main obstacle is cost: every instance requires three successful source executions, environment replay, and three continuations for each of twelve compactors, and this cost grows rapidly with the horizon. We will discuss this direction in the revision and add it to our future work.

Boundaries. We place boundaries at 25%, 50%, and 75% of execution to compare compaction at the early, middle, and late stages of the same tasks under a common protocol. A boundary triggered by the context window would depend on the window and tokenizer of a particular executor. For a given task, it would also yield only late-stage compaction, which confounds execution stage with absolute context length. The stage-controlled design is what reveals that stage effects differ across domains: accuracy decreases toward later boundaries in Search, increases in Coding, and is non-monotonic in Workspace (RQ2). Combined with the fixed 10% retention target, each boundary emulates a point at which accumulated context must be condensed to a fixed budget, whether because the window is full or because context is compacted proactively to control cost. We will state explicitly that our setting is a controlled, stage-based evaluation rather than a window-triggered deployment simulation.

AuthorsOpenReview ↗

Response to reviewer y8rh (part 2)

W2 & Q2: Is the T−bT-b budget fair, and what happens without this constraint?

A2: Thank you for this suggestion and for proposing an alternative evaluation design. We match the post-compaction budget to the number of actions that the successful reference execution used after the boundary. Our question is whether the compacted context supports completing the remaining work as efficiently as the original history did, so that extra search or reconstruction cannot compensate for state lost during compaction. The budget is identical for all compactors, so comparisons among them remain fair. Following your suggestion, we added a relaxed-budget experiment.

Relaxed-budget experiment. For each domain, we selected 10 tasks spanning the three boundaries, covering both continuations that had succeeded and continuations that had exhausted their budget in our main experiments. We reran the same Claude Opus 5 compacted context with a budget of TT actions, i.e., the full length of the reference trajectory, which provides 1.3, 2, and 4 times the matched budget at the 25%, 50%, and 75% boundaries, respectively. Each task was run once.

BenchmarkSucceeded within T−bT-bSucceeded after T−bT-b, by TTUnsuccessful by TT
BrowseComp154
SWE-bench091
CompanyBench361
Total4206

With TT actions, 4 of the 30 continuations finished within T−bT-b actions, 20 succeeded only after using more than T−bT-b actions, and 6 remained unsuccessful. Of these 6, four exhausted the full budget TT, and two terminated before reaching it.

The two settings lead to the same conclusions. This result supports your intuition that many failures under the matched budget are budget failures. The relaxed budget, however, complements rather than contradicts the matched budget: both settings reflect the same compaction failure modes identified in our analysis (RQ4).

  • Compaction loss mostly appears as additional actions. Of the 26 continuations that had not finished within T−bT-b actions, 20 went on to complete the task with additional actions, indicating that these continuations were typically still making progress rather than stuck. This matches our failure analysis: in the main experiments, 91.6%, 92.0%, and 97.7% of failed continuations in Search, Coding, and Workspace, respectively, exhausted their budgets; the executor attempted to recover lost information in 95.1% of applicable failed continuations; and omissions were associated with repeating already completed work. The matched budget therefore captures the cost of re-acquiring or reconstructing state that compaction failed to preserve.
  • Distorted task states are not repaired by additional actions. RQ4 also shows that state distortions, such as premature commitment, lead the executor to act on an incorrect task representation, and that premature commitment is more frequent in early failures than in budget exhaustion. The relaxed budget shows the same pattern: two of the six unsuccessful continuations terminated before exhausting the budget.

Together, the two settings provide complementary views of the same phenomenon: the matched budget measures whether compaction preserves efficient continuation, while the relaxed budget reveals whether the lost state can still be recovered at a higher cost. We will include the relaxed-budget results in the revision as a complementary analysis.

Dear Reviewer y8rh,

Thank you again for your time and your thoughtful review. As the discussion period closes on November 18 (AoE), we wanted to kindly follow up on our response. In brief, we clarified that CompAct-Bench already covers long trajectories averaging 122.8K tokens in Workspace and that our pipeline applies directly to longer-horizon suites (W1 & Q1), explained that stage-based boundaries reveal domain-specific stage effects that window-triggered boundaries would miss (W1), and confirmed with the relaxed-budget experiment you suggested that both budgets lead to the same conclusions (W2 & Q2). We would be grateful to know whether our response addresses your concerns. We would be happy to clarify anything further before the deadline.

AuthorsOpenReview ↗

Gentle Follow-up as Discussion Ends

Dear Reviewer y8rh,

As the discussion period ends in about a day, we would greatly appreciate it if you could let us know whether our responses address your concerns. If they do, we would be grateful if you would consider updating your score. We would also be happy to answer any remaining questions before the deadline.

Reviewer_VzhUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CompAct-Bench, a benchmark for evaluating the ability of long-horizon agents to resume tasks after working-context compression. The authors curate tasks from BrowseComp, SWE-bench and CompanyBench, collect successful execution traces, and insert compression boundaries at 25%, 50% and 75% progress points of execution. A compressor condenses the interaction history before each boundary to roughly 10% of its original character length, after which a fixed executor continues the original task. The work evaluates 12 large language models and carries out further analysis along the following dimensions:

  1. Compression performance across different task domains and execution stages;
  2. The relative contributions of external environment states versus contextual history;
  3. Categories of compression errors;
  4. Qualitative case studies of failure behaviors after compression. The core conclusion of this paper is that existing large models still suffer substantial performance degradation under working-context compression. Compression quality depends not only on the length of history, but also on whether task states have been persisted into the environment.
Critical strengths
  1. Long-horizon agents inevitably has context-window limitations. Therefore, investigating compression quality by downstream task success is important.
  2. The benchmark construction pipeline is clear. The proposed Iterative Ratio Calibration (IRC) is a sensible experimental control design to mitigate the difficulty of precisely controlling output length across different models.
  3. The results are reasonable. For instance, even the best-performing model only achieves an overall success rate of 68.5%; Performance on the Search setting declines as the compression boundary moves forward; Code states persisted in the environment substantially reduce reliance on historical context.
Critical weaknesses
  1. Several critical experimental analyses are missing. For example, the workspace results in Figure 2 show a non-monotonic pattern. What explains this trend? Why does Seed 2.1 Pro drop to 0 accuracy at the 50% compression boundary while achieving higher accuracy at 25% and 75%?
  2. How many original tasks are there in BrowseComp, SWE-bench and CompanyBench before filtering? What fraction of tasks is retained in the final benchmark?
  3. The paper mainly uses 12 LLMs as compressors but lacks thorough comparisons against alternative compression strategies. It is unclear whether the proposed approach outperforms simple baselines such as retaining the most recent turns, plain truncation, or existing context compression methods including extractive compression and structured memory techniques.
  4. The annotation pipeline relies on automatic model labeling, which may introduce noise. In Section 3.5 and Appendix G, GPT-5.6-Sol is used to label compression errors and subsequent behaviors. Only a subset of these labels is manually verified, yet key details are not reported: the number of manually checked samples, the sampling strategy, and the inter-annotator agreement between human and model labels.
Questions for authors

Please refer to weakness.

4 comments
AuthorsOpenReview ↗

Response to reviewer VzhU (part 1)

Thank you for your careful review and for recognizing the importance of evaluating compaction through downstream task success, the clarity of the benchmark construction pipeline, and the Iterative Ratio Calibration design. We address each concern below.

W1: Why are the Workspace results non-monotonic, and why does Seed 2.1 Pro drop to 0% at the 50% boundary?

A1: Thank you for pointing out this pattern. Workspace contains both task progress persisted in the environment and information that remains only in the textual history. As execution proceeds, more progress becomes recoverable from the workspace, while the history requiring compaction also becomes longer. The 50% peak is therefore consistent with a balance between these two effects.

Seed 2.1 Pro’s 0% at the 50% boundary appears to break this pattern, so we additionally examined its Workspace continuations at the requirement level. Our main results count a continuation as successful only under strict end-to-end success, which requires every task requirement to be met. Using the native evaluator’s per-requirement output, we checked whether each continuation opened the pull request in every required repository, passed the QA tests, and completed each required closeout update (ticket, Slack, and Notion). Under this accounting, Seed 2.1 Pro scores 16.7%, 22.7%, and 17.5% at the 25%, 50%, and 75% boundaries, peaking at the midpoint like other compactors, and its continuations reach the PR stage most often at 50% (4/9, versus 1/8 and 2/8). The strict 0% therefore does not reflect a midpoint-specific collapse.

To understand why these continuations still fail, we further compared Seed’s compacted contexts with the corresponding pre-boundary histories and found that they often present unverified work as verified. This is consistent with our error annotation: 64% of Seed’s Workspace compacted contexts contain claims unsupported by the pre-boundary history, the highest among the twelve compactors, and Seed also has the highest rate of premature commitment. In one 50% task, the history ended before any compilation or test had succeeded, yet Seed’s context reported the new migration as successfully applied and the filter behavior as checked. The continuation accepted this status, skipped real testing, opened the PR, and completed every closeout update, but failed the QA tests. Under strict scoring, a single incorrect status fails the whole task even when most requirements are met. This mirrors RQ4, where state distortions lead the executor to act on an incorrect view of what has been completed.

We will add the requirement-level breakdown alongside strict success, together with group sizes and uncertainty estimates. Strict end-to-end success remains our primary metric because deployment requires complete delivery, while the breakdown shows that Seed’s 50% result reflects unreliable status reporting rather than a stage-specific effect.

W2: How many source tasks are there before filtering, and what fraction is retained?

A2: Thank you for asking for the complete filtering statistics. BrowseComp, SWE-bench Verified, and CompanyBench contain 1,266, 500, and 160 source tasks, respectively. Following the protocol described in the paper, we run the fixed executor three times, retain tasks solved in all three runs, and remove trajectories with fewer than eight rounds. This leaves 190 Search, 231 Coding, and 59 Workspace tasks, corresponding to 15.0%, 46.2%, and 36.9% of the respective source pools. We then fix the evaluation size at 200 tasks and retain approximately the same proportion of each eligible pool—41.6%, 41.6%, and 42.4%—while balancing trajectory lengths across the three boundary groups. This yields the final 79, 96, and 25 tasks. Relative to the complete source pools, the final retention rates are 6.2%, 19.2%, and 15.6%. The differences mainly reflect how many tasks pass the solvability and trajectory-length screening in each domain, rather than unequal downsampling after screening. We will add these counts and proportions to the revision.

AuthorsOpenReview ↗

Response to reviewer VzhU (part 2)

W3: How does LLM-based compaction compare with simple and existing compression baselines?

A3: Thank you for suggesting these additional comparisons. Because each strategy requires a fresh downstream continuation, we evaluate the baselines on a cost-controlled subset of 24 task-boundary instances, including 8 Search, 8 Coding, and 8 Workspace instances and covering all three compaction boundaries. Evaluating five baselines on this subset requires 120 additional continuations.

The baselines include recent-turn retention, prefix truncation, head-and-tail retention, BM25-based turn extraction, and rule-based structured state. All methods use the same 10% character budget and the same boundary state, executor, evaluator, and T−bT-b continuation budget. Claude Opus 5 model-generated compaction succeeds on 11/24 tasks, compared with 7/24 for BM25, 5/24 each for recent turns, prefix truncation, and head-and-tail retention, and 4/24 for structured state. These results suggest that relevance-based extraction is useful, but fixed heuristics do not transfer consistently across task domains, while model-generated compaction achieves the highest overall task completion. We will add the complete baseline definitions and results to the revision.

W4: How many annotations were checked by humans, how were they sampled, and how well do human and model labels agree?

A4: Thank you for noting that the description of human validation was incomplete.

Sample. Using a fixed random seed, we sampled 10 of the 200 task–boundary instances (5%), stratified by task domain and compaction boundary, and validated the annotations of all 12 compactors on each instance, giving 120 compaction cases. These instances contain 279 inventory units, i.e., the pieces of information that the boundary-state inventory lists as required for continuation.

Protocol. Every case was labeled independently by two annotators using the same rubric as the model annotator. Annotators did not see the model labels, the compactor identity, or the task outcome.

Agreement. Overall raw agreement was 95% between the two annotators and 85% between the model and the human annotators. Human annotators thus apply the rubric consistently. The model labels are noisier than those of a second annotator, so we interpret them in aggregate, as RQ4 does when comparing category frequencies across compactors and domains. We will add the sampling procedure, annotation protocol, and agreement results to the revision.

Dear Reviewer VzhU,

Thank you again for your time and your thoughtful review. As the discussion period closes on November 18 (AoE), we wanted to kindly follow up on our response. In brief, we explained the Workspace trend and showed that Seed 2.1 Pro also peaks at the 50% boundary under requirement-level scoring (W1), reported the complete filtering statistics (W2), added five baselines and found that model-generated compaction outperforms all of them (W3), and detailed our human validation of 120 stratified cases with 95% inter-annotator and 85% model-human agreement (W4). We would be grateful to know whether our response addresses your concerns. We would be happy to clarify anything further before the deadline.

AuthorsOpenReview ↗

Gentle Follow-up as Discussion Ends

Dear Reviewer VzhU,

As the discussion period ends in about a day, we would greatly appreciate it if you could let us know whether our responses address your concerns. If they do, we would be grateful if you would consider updating your score. We would also be happy to answer any remaining questions before the deadline.

Reviewer_Cs3BOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CompAct Bench, a benchmark for evaluating LLM based working context compaction through downstream task completion across 200 long horizon agent tasks in three domains.

Critical strengths

The paper addresses an important practical problem and proposes a meaningful execution based evaluation framework, with insightful analyses of task state persistence and compaction failure modes.

Critical weaknesses

The continuation budget may confound compaction quality with execution efficiency. The benchmark limits continuation to the remaining actions in the reference trajectory. However, a compacted context may preserve sufficient information while leading the executor to follow a different valid path requiring more actions. Although the authors justify this budget and analyze its effects, the metric may still conflate information preservation with execution efficiency. The generalizability of compaction quality across executors remains unclear. The benchmark evaluates compacted contexts using a fixed executor for each domain. While this ensures controlled comparisons, different executors may interpret the same compacted context differently. It remains unclear whether the resulting model rankings reflect general compaction capability or compatibility with the selected executors. The artificial compaction boundaries may not reflect realistic context window pressure. The benchmark introduces compaction at fixed execution stages regardless of actual context length. For example, Coding trajectories average only 8.8K tokens, which may not require compaction in practice. Although this design supports controlled evaluation, its relevance to real context window constraints remains unclear. Single compaction evaluations may not reflect reliability over repeated compaction cycles. The benchmark evaluates one compaction event per task, whereas long running agents may undergo multiple compactions. Information loss or distortion may accumulate across successive cycles, potentially affecting task completion. It remains unclear whether performance under a single compaction event generalizes to repeated compaction scenarios. The lack of a full history continuation baseline limits the attribution of failures to compaction. Although the authors control task solvability and execution variability, they do not report continuation performance using the complete original history at the same checkpoint and action budget. Without this baseline, it is difficult to distinguish failures caused by compaction from those arising from execution variability or budget constraints, potentially overstating the degradation attributable to compaction.

Questions for authors

Please refer to the questions I raised in the "weakness" section.

5 comments
AuthorsOpenReview ↗

Response to reviewer Cs3B (part 1)

Thank you for your careful review and for recognizing the practical importance of the problem, the execution-based evaluation framework, and our analyses of task-state persistence and compaction failure modes. We address each concern below.

W1: Does the matched continuation budget conflate information preservation with execution efficiency?

A1: Thank you for raising this point. We agree that the matched budget evaluates not only whether the compacted context contains the required information, but also whether it lets the executor complete the task as efficiently as the original history did. This is intentional: in long-horizon deployment, state that has to be re-acquired after compaction costs additional actions, tokens, and time. We therefore set the continuation budget to T−bT-b, the number of actions that the successful reference execution used after the boundary, so that extra search or reconstruction cannot compensate for state lost during compaction. The budget is identical for all compactors, so comparisons among them remain controlled.

The budget does not require the executor to replay the original path. Successful continuations used a median of 55%, 86%, and 93% of the remaining budget in Search, Coding, and Workspace, respectively, and 48% of successful Search continuations used at most half of it.

To separate information sufficiency from execution efficiency, we reran 10 tasks per domain, spanning the three boundaries, with the same Claude Opus 5 compacted context and a relaxed budget of TT actions, which corresponds to 1.3, 2, and 4 times the matched budget at the 25%, 50%, and 75% boundaries. In each domain, we included six tasks whose main-experiment continuation had exhausted the budget and four whose continuation had succeeded. Each task was run once.

DomainPreviously exhausted the budget: succeeded with TTPreviously succeeded: succeeded again with TTOf these, used more than T−bT-b actions
Search4/62/41/2
Coding5/64/44/4
Workspace6/63/42/3
Total15/189/127/9
  • Most budget failures reflect state that can be recovered at a higher cost. 15 of the 18 previously budget-exhausted continuations succeed with TT actions, and 3 still fail. This matches our failure analysis: 91.6%, 92.0%, and 97.7% of failed continuations in Search, Coding, and Workspace exhaust their budgets, the executor attempts to recover lost information in 95.1% of applicable failed continuations, and critical omissions are associated with repeating completed work. The additional actions are therefore the cost of re-acquiring state that compaction did not preserve.
  • Removing the cap changes what is measured. Of the 9 previously successful continuations that succeeded again with TT actions, 7 used more than T−bT-b actions. Given more budget, the executor tends to keep searching or testing before it submits, even on tasks it had completed within the matched budget. Success and action counts under a larger budget therefore depend on the budget itself, whereas a matched cap gives every compactor the same reference.

In the revision, we will state explicitly that the primary metric measures task completion under the matched budget, and we will report relaxed-budget success and the number of actions used as complementary metrics.

W2: Do the rankings reflect general compaction ability or compatibility with the chosen executors?

A2: Thank you for raising this important distinction. CompAct-Bench is designed to compare compactors while holding downstream execution conditions fixed. Within each domain, we therefore use the same executor, tools, environment, and evaluator, leaving MbM_b as the primary variable across models. If the same compacted context were given to different executors, the outcome would also reflect differences in their task-solving ability and interpretation of the context, constituting a complementary cross-executor transfer evaluation. Although the current benchmark does not isolate this dimension, our experiments span three domains and two execution systems, and the positive cross-domain rank correlations provide preliminary evidence of shared compaction ability. We will clarify that the reported scores represent controlled comparisons under a given executor.

AuthorsOpenReview ↗

Response to reviewer Cs3B (part 2)

W3: Do fixed compaction boundaries reflect realistic context-window pressure?

A3: Thank you for raising the question of practical relevance. When an agent performs compaction depends on both the model’s context window and the system’s compaction policy, so CompAct-Bench does not attempt to reproduce one fixed context-overflow point. We apply compaction at 25%, 50%, and 75% task progress to compare how compaction during the early, middle, and late stages affects subsequent task completion. Search and Workspace provide longer-context settings, with complete trajectories averaging 35.3K and 122.8K tokens. Coding trajectories average 8.8K tokens and may fit within a larger context window. Here, they primarily represent proactive compaction to reduce context-processing cost rather than compaction forced by a hard window limit. We will revise the statement that each boundary represents reaching the context-window limit and instead clarify that these boundaries simulate compaction at different task stages, whether motivated by window pressure or proactive cost control. CompAct-Bench therefore evaluates whether the task can still be completed after compaction, rather than prescribing when compaction should be triggered.

W4: Does performance under a single compaction generalize to repeated compaction?

A4: Thank you for highlighting repeated compaction as an important setting. We intentionally begin with a single compaction to provide a clean evaluation setting that directly measures how one context replacement affects subsequent task completion. Even a single compaction separates compactors widely: overall accuracy ranges from 46.0% to 68.5%, and the bootstrap confidence intervals of the strongest and weakest compactors do not overlap ([62.0, 74.5] vs. [40.0, 52.5]). In Coding, replacing the history with one compacted context lowers accuracy by up to 21.7 points relative to continuing from the same checkpoint with the complete history (W5). This shows that a single compaction is already a challenging and meaningful capability to evaluate on its own. Accordingly, our current results primarily characterize the reliability of a single compaction. Introducing multiple cycles at once could accumulate omissions or distortions from earlier cycles and make the final failure difficult to attribute to a particular compaction. Repeated compaction therefore requires finer-grained tracking of the task state and subsequent execution after each cycle. Our resumable execution pipeline supports this extension by allowing additional compaction boundaries and evaluation after each cycle. We plan to build on this capability to extend CompAct-Bench to repeated compaction and will add it as an important future evaluation direction in the revision.

AuthorsOpenReview ↗

Response to reviewer Cs3B (part 3)

W5: Can failures be attributed to compaction without a full-history continuation baseline?

A5: Thank you for suggesting this control. We added a full-history continuation baseline that matches the compaction setting in every respect except the context: the executor receives the complete pre-boundary history HbH_b, including all tool calls and results, and continues from the same boundary state under the same T−bT-b budget. We randomly sampled 20 Search and 20 Coding tasks, stratified by boundary and remaining budget and selected without reference to any outcome. We compared the full history with compacted contexts from three compactors spanning the range of Table 1: Claude Opus 5 (strongest), HY3 (median), and Seed 2.1 Pro (weakest). Each condition was run three times per task, giving 480 continuations. Workspace is omitted because its harness carries context over only as text in the task instruction and cannot restore the original tool-call history.

Strict success (%)SearchCodingOverall
Full history HbH_b83.386.785.0
Claude Opus 555.078.366.7
HY351.765.058.3
Seed 2.1 Pro35.073.354.2

The results lead to three conclusions.

The matched budget is achievable from the reconstructed checkpoints. With the complete history, the executor completed 85.0% of continuations within T−bT-b actions (83.3% in Search and 86.7% in Coding). The checkpoints and the matched budget are therefore sufficient for continuation in most cases, so post-compaction failures cannot be explained mainly by an infeasible checkpoint, executor variance, or an insufficient budget.

Replacing HbH_b with MbM_b causes a compactor-dependent loss. Under otherwise identical conditions, overall accuracy drops by 18.3, 26.7, and 30.8 points for Claude Opus 5, HY3, and Seed 2.1 Pro, respectively. The ordering of the three compactors matches their overall ordering in Table 1, so the gap to the full-history reference reflects the quality of the compacted context rather than a shared ceiling.

The loss is larger in Search than in Coding, consistent with RQ3. In Search, where gathered evidence and search progress remain largely context-resident, compaction lowers accuracy by 28.3 to 48.3 points. In Coding, where much of the progress persists in the repository, the drop is 8.3 to 21.7 points. This matches our finding that the utility of compacted context depends on how task state is distributed between the environment and the working context.

Together, these results address the attribution concern directly: from the same checkpoint and under the same budget, the executor usually completes the task with the complete history, and the gap between this reference and each compactor isolates the loss attributable to compaction. We will add this baseline to the revision and report the full-history reference alongside the main results, so that failures shared with the full-history condition are not attributed to compaction.

Comments on the paper

1 comment
Area_Chair_hjJcOpenReview ↗

Review response

Dear Reviewers, the authors have responded to your reviews with additional experiments, and the author-reviewer discussion ends soon. Please read their response, post a brief reply on whether your concerns are addressed, and update your scores if your assessment has changed. Thank you! AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.