acceptodds
Under review as a conference paper at ICLR 2027

Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents

Abstract

As LLM agents tackle longer tasks, they increasingly compress growing histories of reasoning, actions, and tool outputs. Compression can reduce token use, but it also changes the information available for later decisions. Existing agentic harnesses bundle decisions about what to compress, when to compress, and how much to remove into fixed policies. A systematic characterization is needed to disentangle these decisions and reveal how each affects task success and execution cost. We systematically vary these decisions across three open-weight models on SWE-bench Verified and Terminal-Bench 1.0. Across nearly 35,000 agent runs, we measure task success, token use, end-to-end latency, and estimated cost. We find that fewer tokens need not mean faster or cheaper execution: on Terminal-Bench with Qwen, policies using roughly one-third as many tokens can take 20–80% longer than the uncompressed agent. Policies with similar overall success can solve different tasks, while the same policy can perform quite differently across models. Our results motivate evaluating compression by its effects on agent execution and tailoring policies to the task, model, and workload.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gmkGOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a systematic empirical study of context compression in LLM agents. It decomposes compression policies into three dimensions: the compression primitive, the trigger, and the compression depth. The authors evaluate 65 policy configurations across three open-weight models on subsets of SWE-bench Verified and Terminal-Bench, conducting nearly 35,000 agent runs. The evaluation measures task success, token consumption, end-to-end latency, and estimated input billing cost. The main findings are that reducing token consumption does not necessarily reduce latency or cost, policies with similar aggregate success rates can solve different tasks, and the effectiveness of a compression policy varies across models and workloads.

Critical strengths

The paper addresses an important problem for long-running LLM agents: how context compression affects task success and execution efficiency beyond token savings.Decomposing compression policies into primitives, triggers, and depths provides a useful structure for organizing and comparing context-management strategies.The study evaluates 65 policy configurations across three open-weight models and two coding benchmarks, with repeated runs totaling nearly 35,000 agent executions.

Critical weaknesses
  1. Limited novelty and incremental contribution. The effects of context compression on model reasoning, task success, and execution efficiency have already been discussed in prior research and publicly documented industrial practice, with some studies also proposing mitigation strategies. The paper does not sufficiently establish what substantive new insights its central findings provide beyond this existing understanding.

  2. Limited actionable guidance and insufficient support for some conclusions. The paper identifies performance trade-offs among compression policies but does not propose and validate targeted improvements or concrete criteria for policy selection. Some experimental results exhibit substantial uncertainty, and the conclusions remain largely descriptive, providing insufficient support for clear and generalizable practical recommendations.

  3. Insufficiently operationalized quantitative evaluation criteria. Although the paper reports success rate, token consumption, latency, and estimated cost, it does not establish clear criteria for comparing policies under matched resource budgets or comparable task success rates. Nor does it adequately define acceptable trade-offs between performance degradation and efficiency gains. Consequently, the evaluation provides limited support for reliable policy ranking and deployment decisions.

Questions for authors

Can the authors quantify the contributions of summarization overhead, cache invalidation, and information reacquisition to the observed cost and latency changes, and use this analysis to validate a mitigation strategy under matched resource budgets or comparable task success rates?

3 comments

R3W1: Novelty over prior work. To our knowledge, no prior study isolates the individual choices that make up a compression policy and measures their effects under controlled conditions. Existing work either proposes a method and evaluates it as a complete system (Kang et al., 2025; Chen et al., 2026; Lindenbauer et al., 2025; Xiao et al., 2026), or observes a deployed mechanism in production (Liu et al., 2026; Tang et al., 2026). In both cases the mechanism, trigger and depth change together, so an observed effect cannot be assigned to any one of them (§A.1). This is the gap our team identified and show through this detailed study. We hold the harness, tasks and model fixed and vary one choice at a time, across 65 policies, three models and two benchmarks. This yields findings that prior work could not produce. Token savings do not carry over to cost, since policies using 0.6x the tokens still cost 0.71-0.95x (Obs 2). Policies with matching resolve rates solve different tasks (Obs 4). Policy rankings reverse across models, with one policy gaining 7.0 points on Qwen and losing 17.0 on Devstral (Observation 5). We would welcome pointers to specific studies the reviewer has in mind, and we will add a direct comparison with them.

R3W2: Actionable guidance. Our study gives the community three things it did not have. The first is a common design space. Primitive, trigger and depth describe every deployed mechanism we surveyed, so new methods can be compared on the same terms. The second is an evaluation standard. We show that token savings do not predict cost or latency, and that equal resolve rates can hide different solved tasks. A method evaluated on tokens and aggregate success alone can therefore look better than it is. The third is a reference set of 65 policies across three models and two benchmarks, which any new method can be measured against. The study also gives guidance that can be used today, which previous work do not establish. Keeping recent history verbatim and clearing tool outputs before summarizing each improves SWE-bench resolve rate by 6.7 to 9.7 points on all three models, with every 95% interval excluding zero (Figure 5b). It also identifies open problems for future work, such as why policy rankings reverse across models and how to compress without invalidating the prompt cache.

R3W3: Evaluation criteria. We would be glad to address this more precisely if the reviewer could point to the specific comparisons they found unclear. In the meantime, we note that the paper already supports matched comparisons. Figure 2 plots each policy’s success against its token use, latency and cost, so policies can be read at equal token use or equal success. We report these metrics separately because the acceptable trade depends on the deployment, and a single score would hide it. Where an explicit rule is wanted, dominance provides one, and tool-result clearing is never dominated across our six model and benchmark settings.

Thank you for the clarification. I recognize the value of the study’s experimental scale and unified evaluation, and I acknowledge the paper’s discussion of cache reuse, execution length, and compression overhead. My remaining concern is the extent to which these observations provide new explanatory insights beyond what is already known. Taking Observation 2 as an example, prior research has provided mechanistic analyses and ablation evidence showing that token count alone does not determine execution efficiency. For instance, SGLang (Zheng et al., NeurIPS 2024) demonstrates how cache hit rates and scheduling strategies affect latency and throughput through controlled ablations. Therefore, observing that token savings do not translate into proportional reductions in cost or latency is not, by itself, sufficient to establish insights that prior work could not have produced. I would appreciate further clarification of what new relationships the paper’s controlled experiments identify beyond these known mechanisms, and which attributions are supported by independent experimental evidence. Similarly, task performance after compression depends on the relevance of the retained information to the task, rather than token count alone. Ablations in LongLLMLingua (Jiang et al., ACL 2024) show that removing the question-aware fine-grained compression module substantially reduces task performance even at comparable token counts.

We thank the reviewer for their response and concrete references. Relation to SGLang. We build on SGLang and cite it for exactly this mechanism (§A.1, “Serving efficiency and end-to-end cost”). The critical difference is what is held fixed. SGLang replays recorded agent traces and disables any optimization that would change the computation results, so every system serves the same calls. It contains no context compression. We vary the compression policy and show that the policy sets the quantities SGLang takes as inputs. Each policy changes how much of the prefix is reused, how many calls the agent makes, and how much it re-reads.

Relation to LongLLMLingua. We agree that relevance matters more than token count. LongLLMLingua scores relevance against a known question and evaluates single-answer QA tasks. An agent compresses mid-task, before it knows what later steps will be needed. Our contrasts show which structural choices protect that information. Keeping recent turns verbatim adds 6.7–9.7 points on SWE-bench across three models.

Independent evidence. Each attribution rests on a direct measurement. Step counts are logged per run and Cache reuse is checked. Design effects come from contrasts that change one choice.

What we want the community to take away. Compression in an agent is not a preprocessing step that shrinks a prompt. It is a policy that changes what the agent does next, so it should be judged by the full execution it induces and not just by tokens removed. This is critical because of how agents are used today. Open-source agents such as OpenHands ship one default policy (Table 2), and users attach whichever model they prefer. These users see the token savings immediately, but they do not see what the savings cost in task success and latency. Our results show that this cost is real and that it changes with the model. A policy that works well with one model can lose a large share of task success with another, and nothing in the agent warns the user when this happens. This is why we argue that a compression policy should be validated for each model and not inherited from a default.

Reviewer_aWUzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a counter-intuitive finding that compressing tokens in LLM context does not always leads to improved efficiency of tasks, and proposes that compression should be evaluated more comprehensively on different models and tasks to show their effectiveness instead of just counting the saved tokens.

Critical strengths

This paper proposes a quite interesting perspective of analyzing compression effectiveness, while most of them directly reduces tokens, their effectiveness on different tasks, models, could be very different.

Critical weaknesses
  1. The paper studies a number of compression methods with defining its primitive, trigger, and depth - which I would question (1) whether this reflects a global picture of how everyone is doing compression, and (2) whether the compression methods in comparison are really representative for thousands of different compression methods. For example, the compression can be really complex and are using different strategies dynamically at different step, and these compression shall be already optimized with engineers. Claude Code enables users to compress the context at any time by using a command line instruction, and I do not think the evaluated compression mechanisms reflect the state of the art compression mechanisms.

  2. The findings of compression on different tasks and models shall perform quite differently is not really surprising. While i acknowledge observation # 2 as an interesting one, the paper neither proposes a compression method or evaluation framework for any compression strategies. So for me, this is is mostly an analytical paper. The title claiming "A Systematic Study of Context Compression" is a bit too overwhelming.

Questions for authors

I wonder whether the authors could demonstrate some of the compression strategies that is state-of-the-art and widely adopted by frontier labs are really as simple as the compression defined in this paper (with primitive, trigger, and depth only), and whether the analysis results are consistent.

2 comments

R2W1 & R2W2 & Q1: Coverage of deployed compression. We agree with the reviewer that deployed systems are more elaborate than any single policy in our grid. They bundle several choices together. Claude Code, for example, clears tool outputs, summarizes the full history, and fires near the context limit, all as one mechanism. This bundling is the reason for our study. When a bundled system succeeds or fails, one cannot tell whether the cause is the clearing step, the summary, the trigger point, or how much history is kept. As §1 notes, such end-to-end evaluations are difficult to interpret. To our knowledge, no vendor has published a comparison of these alternatives. We therefore take the building blocks from deployed systems (§2.1) and vary them one at a time in a common harness. The building blocks do cover what is deployed. Table 2 maps OpenHands, Gemini CLI and Claude Code onto our three axes. We have since surveyed 17 deployed agents, including Codex, Copilot, Cursor and Cline. The default of each one is a combination of our primitives with a threshold trigger. A manual command such as ‘/compact’ runs the same primitive at a time the user chooses. Separating the choices also tells us things a bundled evaluation cannot. In Observation 6, clearing tool outputs before a structured summary, which is the design Claude Code ships, improves resolve rate by 7.7 to 9.7 points over the summary alone on all three models. We will expand Table 2 to the full survey and state this motivation more directly in §1.

Thanks for the response, I will keep my recommendation given the response.

Reviewer_N23gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how compression primitive, trigger, and depth affect LLM-agent execution. Its contributions are a common framework for comparing compression policies and a large empirical characterization: reportedly nearly 35,000 runs across three open-weight models, with main comparisons on 100 SWE-bench Verified issues and 40 Terminal-Bench tasks, plus smaller parameter sweeps. Validation uses benchmark-native evaluators, three attempts per configuration, and task-bootstrap confidence intervals. The main findings are that token savings need not improve latency or estimated input cost, and that policy rankings and solved-task sets vary across models and workloads.

Critical strengths
  1. A useful, systematic experimental framework. Varying compression choices within a common agent harness makes the comparisons substantially more informative than evaluating isolated, bundled systems. The breadth across policies, models, and environments makes this a valuable empirical reference for agent developers.
  2. Concrete findings with practical consequences. For example, partial structured summarization improves SWE-bench resolve rate over full structured summarization by 6.7, 8.7, and 8.0 percentage points across the three models (Figure 5). Such consistent differences demonstrate that preservation choices materially affect task completion and provide actionable guidance beyond minimizing context length.
Critical weaknesses
  1. The efficiency comparisons exclude a consequential, policy-dependent portion of execution. Main resource metrics omit timed-out runs (§A.5), although timeouts are substantial: on Qwen/Terminal-Bench, full summarization times out in approximately 46% of attempts versus 21% for full context (Figure 14). Pairing tasks does not eliminate selection among attempts within each task. Additionally, cost uses reconstructed cache reuse and excludes output charges, while Qwen latency is measured with prefix caching disabled (§A.5.2). These disclosed choices limit whether the reported cost–latency tradeoffs represent the resources required for the complete workload.
  2. The claimed separation of compression choices is insufficiently validated in execution. Appendix A.2 specifies soft summary-length targets with a 4,096-token output cap, tool-result clearing that can fall back to truncation, and partial-summary fallbacks that can delete the summary itself. Without achieved-depth distributions and fallback frequencies, nominal policy settings may describe materially different operations or fail to produce distinct compression depths. This weakens attribution of observed effects to the paper’s central design axes.
Questions for authors
  1. What compression depths were actually achieved, and how frequently did summaries hit the output cap or policies invoke truncation fallbacks?
  2. Do the main efficiency rankings persist when timed-out attempts are included, and what validation supports the reconstructed cache-reuse estimates?
1 comment

R1W1: Timed out runs. As described in §A.5, we exclude timed-out runs only from the resource ratios, because a capped run records the time limit and not the cost of finishing. This follows SWE-agent, which averages cost over successfully resolved tasks (Yang et al., 2024). Timed-out runs still count as failures in resolve rate (Figure 14). We agree that including them is also reasonable, and doing so, as in Figure 3’s table, leaves SWE-bench rankings largely intact (Spearman 0.79–1.00). Rankings also hold when we keep only tasks without timeouts (Qwen and Devstral, 0.85-0.99). Output charges. Pricing output at 4-5x the input rate leaves SWE-bench cost rankings nearly unchanged (0.92-0.94). Prefix Caching. We would like to clarify §A.5.2. Prefix caching was off only for Qwen’s SWE-bench runs, which followed the vLLM default for hybrid-attention models. Qwen’s Terminal-Bench runs and all Devstral and GLM runs were served with caching on. The Terminal-Bench latencies in Figure 2 and the Devstral and GLM latencies in Figure 5a therefore already reflect caching. We also repeated Qwen’s SWE-bench comparison with caching on (975 runs). The measured cache hits reproduce our cost ratios within 0.11 for threshold policies.

R1W2: Separation of compression choices. Our three axes separate the design choices, and each policy is then run as a complete, working mechanism. Fallbacks are necessary for this. A policy must always return a context that fits, and clearing tool results or requesting a shorter summary cannot guarantee that alone. Deployed systems make the same choice and chain several reduction stages (§2.1, and Table 2 in §A.2.2). The paper also defines depth as a target size and not a guarantee (§2.3), and §A.2 documents each fallback the reviewer lists. The main comparisons (Figures 2, 4 and 5) use one fixed depth, so they do not depend on depth separation. Achieved depth distributions & fallback frequencies. We agree these should be reported, and we have now measured every compression event. Truncation and the two partial summarizers reach the depth we set. Truncation, for example, keeps 71%, 51% and 30% of the prompt at D = 0.3, 0.5 and 0.7. Full summarization follows depth only when the requested summary is short. On Terminal-Bench the requests are short, and Qwen’s summaries deepen in step with D. On SWE-bench the requests are several thousand tokens, the models write under one thousand, and every setting ends up removing about 95% of the history. This is a known limit of long length instructions (Zhang et al., 2025). Tool-result clearing falls back to truncation in 57-84% of SWE-bench events. In the revision we will report the full distributions and limit Observation 3’s depth claims on SWE-bench to the three primitives that reach their target. R1Q1 & R1Q2: Q1 (achieved depths, cap hits, fallbacks). Please see our response to W2 above. We promise to add the full distributions to the appendix. Q2 (rankings with timeouts, cache validation). Please see our response to W1 above. SWE-bench rankings persist with timed-out attempts included, and the cache estimate is validated against measured vLLM cache hits on 975 runs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.