- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper systematically investigates the effects of context compaction on long-horizon LLM agents, focusing on downstream task performance and the preservation of user-specified constraints. Using OpenCode as a fixed agent harness, the authors evaluate 4 frontier models across 3 long-horizon benchmarks, comparing different compaction thresholds and five compaction strategies. The results show that compaction thresholds have limited effects on task performance for models with 1M-token context windows, while no alternative strategy consistently outperforms default self-summarization. However, analyses of ProgramBench and real user-agent sessions from SWE-chat reveal that compaction can progressively discard task constraints, particularly instructions about how the agent should work and what it should avoid. The authors further suggest that existing benchmarks may underestimate such risks because task objectives remain fixed and much of the agent's progress can be recovered from persistent files.
Critical strengths
-
Meaningful and practically relevant research topic. Context compaction is widely used in long-horizon LLM agents, yet its effects on downstream task performance and instruction preservation remain insufficiently understood. This paper addresses an important practical question for designing and evaluating agent systems, particularly as agents increasingly execute complex tasks requiring multiple rounds of compaction.
-
Relatively systematic experimental design. The paper evaluates multiple frontier models, compaction thresholds, and compaction strategies under a fixed agent harness, reducing potential confounding factors introduced by different agent frameworks. The experiments also span three distinct long-horizon task settings, providing a broader perspective on the effects of compaction across models and workloads.
-
Insights beyond downstream task performance. Rather than relying solely on final benchmark scores, the paper additionally examines the preservation of task constraints during compaction. The analyses of ProgramBench and real user-agent sessions from SWE-chat reveal that process-related instructions and scope restrictions are more likely to be discarded than feature requirements. This distinction provides a useful perspective on the limitations of existing benchmarks, where stable task performance does not necessarily imply faithful preservation of user instructions.
Critical weaknesses
1. Limited Model Coverage and Compaction Threshold Range. The experiments do not fully explore how compaction affects models with different context window sizes. For the three models with 1M-token context windows, compaction thresholds are limited to 160K–380K tokens, leaving the effects of compaction near the context window limit unexplored. Meanwhile, GPT-5.3-Codex is the only model evaluated with a context window smaller than 1M tokens, making it unclear whether the observed sensitivity to lower compaction thresholds generalizes to other models. These limitations weaken the generalizability of the findings regarding the relationship between compaction thresholds, context window sizes, and downstream task performance.
2. Relatively Small Task Coverage. Although the experiments span three long-horizon benchmarks, the evaluation includes only nine tasks or task configurations in total (3 from PaperBench, 2 from ProgramBench, and 4 from PostTrainBench). Such a limited sample may not adequately capture task-level variability, potentially weakening the reliability and generalizability of the findings. Expanding the evaluation to more tasks, ideally the full benchmark sets where feasible, would provide stronger evidence for the reported conclusions.
3. Limited Analysis of the Downstream Impact of Constraint Erosion. While the paper provides evidence that compaction can progressively discard user constraints and reports some observed instruction violations, the practical consequences of such information loss remain insufficiently explored. In particular, the extent to which constraint erosion directly contributes to subsequent instruction violations or degraded task outcomes remains unclear. A more systematic analysis of agent behavior after compaction could help establish the practical significance of this finding.
Questions for authors
-
Have the authors measured whether the selected tasks would exceed the 1M-token context window without compaction? For tasks that can be completed within this limit, have the authors evaluated a no-compaction baseline? Such a comparison would help clarify whether compaction is necessary or beneficial for these tasks.
-
This paper suggests that existing benchmarks with fixed objectives and persistent files may be relatively insensitive to compaction, while the SWE-chat analysis highlights greater risks of constraint loss in real user-agent interactions. Have the authors considered evaluating how compaction affects downstream task performance in multi-turn interactive settings, where user instructions may evolve over time? It would also be valuable to investigate whether different compaction strategies exhibit different effectiveness in such settings.
2 comments
Response 2/2
Weakness 3 - Downstream impact of constraint erosion.
We added two analyses of our existing runs.
First, a rule-aware re-scoring of our existing runs (App. 6.11, Fig. 11). A run that violates a rule gets a score of 0, and we compare this with the raw score. We can do this on ProgramBench and PostTrainBench. When we bin runs by number of compactions, the pooled raw score stays within 0.07 of the task mean in every bin. The rule-aware score is 0.03 below the task mean with no compaction, 0.14 below at 4 to 7 compactions, and 0.28 below at 32 or more. This shows that the standard score does not register rule violations: runs that violate rules score about as well as the ones that don’t. But on its own, it does not show that compaction causes the violations.
Second, we tested whether rules that a summary drops are broken more often afterwards (App. 6.13). On ProgramBench, a rule that the summary dropped was broken before the next compaction in 23 of 3,940 cases (0.6%). A rule that it kept was broken in 3 of 1,040 cases (0.3%). On SWE-chat, 7 of 24 dropped constraints that came up again were violated (29%), against 26 of 137 preserved ones (19%). Dropped rules are thus violated about twice as often in both settings. The counts are small and neither difference is statistically significant (p = 0.33 and p = 0.28, Fisher exact test), so we do not claim a causal link. The same appendix shows when violations start (Table 4). For Claude Opus 4.6, 10 of 12 violating runs break a rule before any compaction. For the GPT models, almost all first violations come after a compaction, and most follow a summary that had dropped the rule. We corrected the statement in Section 4.2 accordingly. We see the causal question as the main open question raised by our results.
Q1: Would the tasks exceed 1M tokens, and is there a no-compaction baseline?
We didn’t run a separate no-compaction baseline, but Fig. 5 shows accuracy as a function of the number of compactions per run, including runs with no compaction. For the 1M models, runs with no compaction score about the same as runs with one or more compactions. On the benchmarks and models we study, compaction therefore doesn’t seem to affect performance, which supports our argument that current benchmarks are compaction-friendly. Regarding tasks exceeding 1M tokens, we did not measure this directly. At 380K, Claude Opus 4.6 averages 0.71 compactions per run (Table 2), so most of its runs compact at most once and would fit in the 1M window. GPT-5.4 averages 1.96, so its typical run is close to the limit.
Q2: Multi-turn settings with evolving instructions.
We agree, and this is the main conclusion of our paper: such benchmarks are currently missing, and they are where compaction should matter most. We could not create and run a new interactive benchmark within the discussion period, and we list it as future work (Section 5).
Our SWE-chat analysis is the closest setting we could study. It shows that process instructions and scope exclusions given by real users are dropped about 6 times more often than feature requirements.
Whether compaction strategies differ in such settings remains open, but even on our fixed-objective benchmarks, strategies that discard more (removing reasoning blocks, summarizing with a smaller model) already cost some models up to 20 points on PaperBench (Fig. 3), so we expect larger differences when instructions exist only in the context.
Response 1/2
We thank the reviewer for the careful reading and the constructive suggestions. The general response lists what changed in the revised PDF.
Weakness 1 - Model coverage and threshold range.
We agree that the submission did not explain how we chose the thresholds. The variable we meant to vary is the number of compactions per run, because each compaction is an opportunity to lose information. And the threshold is how we control it. We chose thresholds that produce a wide range of compaction counts: for the GPT models at a threshold of 160K we have between 15 to 28 compactions per run on average, and at 380K we have fewer than 2 on average (App. 6.8, Fig. 8, Table 2). Section 4.1 now explains this.
On the effect of the context window size, our existing results show that the fraction of the window does not explain the GPT-5.3-Codex sensitivity. GPT-5.3-Codex is at 160K is 40% of its window, and the 1M models at 380K are 38% of theirs. At this matched fraction, Codex scores 0.38 on PaperBench while Claude Opus 4.6 and GPT-5.4 score about between 0.6 and 0.65. The 1M models are also flat at 16% of their window, a lower fraction than any Codex condition.
Conversely, our range already reaches 95% of the Codex window (380K), and there Codex matches the 1M models. Its degradation appears only at low thresholds, where it compacts most often: Fig. 5 shows Codex declining from about 8 compactions per run onward, while GPT-5.4 does not degrade even at 16 or more. This points to a model-specific fragility to repeated compaction rather than an effect of window size. We agree that a single model with a smaller window cannot show whether this generalizes, and we now state explicitly in the paper that the Codex result may be specific to that model.
For the 1M models, thresholds near 90-95% of the window would rarely trigger compaction on these benchmarks: even at 380K, Opus 4.6 compacts only 0.71 times per run on average and GPT-5.4 1.96 times (Table 2). Such thresholds would therefore mostly measure runs without compaction (see our answer to Question 1)
Weakness 2 - Task coverage.
The reviewer's count of nine was correct for the submission. The revision adds one PostTrainBench target (BFCL), for 11 task configurations in total: 3 PaperBench tasks, 2 ProgramBench tasks and 6 PostTrainBench configurations (2 base models × 3 targets). The constraint-erosion analysis also uses the lua task, and the reward-hacking analysis uses lua and luajit.
We could not afford more, because the cost multiplies. Each task is run under 10 conditions (six thresholds with the default strategy, and four alternative strategies at 160K), for three models with two seeds each. That is about 60 runs per task before reruns of failed submissions. Each run takes 6 to 10 hours and costs 30-50 USD on average in API credits, and around 30-40 USD in GPU time.One more task therefore costs roughly 3,600 USD to 5,400 USD. We chose to cover several frontier models, thresholds and strategies on realistic long-horizon tasks rather than many tasks with fewer conditions.
No strategy beats self-summarization on any of the three benchmarks (Fig. 3). The 1M models are insensitive to the threshold on PaperBench and ProgramBench, while on PostTrainBench the error bars are too wide to conclude either way (Section 4.1). We state our claims as conditional on this task set in the Limitations paragraph.