- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a systematic empirical study of context compression in LLM agents. It decomposes compression policies into three dimensions: the compression primitive, the trigger, and the compression depth. The authors evaluate 65 policy configurations across three open-weight models on subsets of SWE-bench Verified and Terminal-Bench, conducting nearly 35,000 agent runs. The evaluation measures task success, token consumption, end-to-end latency, and estimated input billing cost. The main findings are that reducing token consumption does not necessarily reduce latency or cost, policies with similar aggregate success rates can solve different tasks, and the effectiveness of a compression policy varies across models and workloads.
Critical strengths
The paper addresses an important problem for long-running LLM agents: how context compression affects task success and execution efficiency beyond token savings.Decomposing compression policies into primitives, triggers, and depths provides a useful structure for organizing and comparing context-management strategies.The study evaluates 65 policy configurations across three open-weight models and two coding benchmarks, with repeated runs totaling nearly 35,000 agent executions.
Critical weaknesses
-
Limited novelty and incremental contribution. The effects of context compression on model reasoning, task success, and execution efficiency have already been discussed in prior research and publicly documented industrial practice, with some studies also proposing mitigation strategies. The paper does not sufficiently establish what substantive new insights its central findings provide beyond this existing understanding.
-
Limited actionable guidance and insufficient support for some conclusions. The paper identifies performance trade-offs among compression policies but does not propose and validate targeted improvements or concrete criteria for policy selection. Some experimental results exhibit substantial uncertainty, and the conclusions remain largely descriptive, providing insufficient support for clear and generalizable practical recommendations.
-
Insufficiently operationalized quantitative evaluation criteria. Although the paper reports success rate, token consumption, latency, and estimated cost, it does not establish clear criteria for comparing policies under matched resource budgets or comparable task success rates. Nor does it adequately define acceptable trade-offs between performance degradation and efficiency gains. Consequently, the evaluation provides limited support for reliable policy ranking and deployment decisions.
Questions for authors
Can the authors quantify the contributions of summarization overhead, cache invalidation, and information reacquisition to the observed cost and latency changes, and use this analysis to validate a mitigation strategy under matched resource budgets or comparable task success rates?
3 comments
Thank you for the clarification. I recognize the value of the study’s experimental scale and unified evaluation, and I acknowledge the paper’s discussion of cache reuse, execution length, and compression overhead. My remaining concern is the extent to which these observations provide new explanatory insights beyond what is already known. Taking Observation 2 as an example, prior research has provided mechanistic analyses and ablation evidence showing that token count alone does not determine execution efficiency. For instance, SGLang (Zheng et al., NeurIPS 2024) demonstrates how cache hit rates and scheduling strategies affect latency and throughput through controlled ablations. Therefore, observing that token savings do not translate into proportional reductions in cost or latency is not, by itself, sufficient to establish insights that prior work could not have produced. I would appreciate further clarification of what new relationships the paper’s controlled experiments identify beyond these known mechanisms, and which attributions are supported by independent experimental evidence. Similarly, task performance after compression depends on the relevance of the retained information to the task, rather than token count alone. Ablations in LongLLMLingua (Jiang et al., ACL 2024) show that removing the question-aware fine-grained compression module substantially reduces task performance even at comparable token counts.
We thank the reviewer for their response and concrete references. Relation to SGLang. We build on SGLang and cite it for exactly this mechanism (§A.1, “Serving efficiency and end-to-end cost”). The critical difference is what is held fixed. SGLang replays recorded agent traces and disables any optimization that would change the computation results, so every system serves the same calls. It contains no context compression. We vary the compression policy and show that the policy sets the quantities SGLang takes as inputs. Each policy changes how much of the prefix is reused, how many calls the agent makes, and how much it re-reads.
Relation to LongLLMLingua. We agree that relevance matters more than token count. LongLLMLingua scores relevance against a known question and evaluates single-answer QA tasks. An agent compresses mid-task, before it knows what later steps will be needed. Our contrasts show which structural choices protect that information. Keeping recent turns verbatim adds 6.7–9.7 points on SWE-bench across three models.
Independent evidence. Each attribution rests on a direct measurement. Step counts are logged per run and Cache reuse is checked. Design effects come from contrasts that change one choice.
What we want the community to take away. Compression in an agent is not a preprocessing step that shrinks a prompt. It is a policy that changes what the agent does next, so it should be judged by the full execution it induces and not just by tokens removed. This is critical because of how agents are used today. Open-source agents such as OpenHands ship one default policy (Table 2), and users attach whichever model they prefer. These users see the token savings immediately, but they do not see what the savings cost in task success and latency. Our results show that this cost is real and that it changes with the model. A policy that works well with one model can lose a large share of task success with another, and nothing in the agent warns the user when this happens. This is why we argue that a compression policy should be validated for each model and not inherited from a default.
R3W1: Novelty over prior work. To our knowledge, no prior study isolates the individual choices that make up a compression policy and measures their effects under controlled conditions. Existing work either proposes a method and evaluates it as a complete system (Kang et al., 2025; Chen et al., 2026; Lindenbauer et al., 2025; Xiao et al., 2026), or observes a deployed mechanism in production (Liu et al., 2026; Tang et al., 2026). In both cases the mechanism, trigger and depth change together, so an observed effect cannot be assigned to any one of them (§A.1). This is the gap our team identified and show through this detailed study. We hold the harness, tasks and model fixed and vary one choice at a time, across 65 policies, three models and two benchmarks. This yields findings that prior work could not produce. Token savings do not carry over to cost, since policies using 0.6x the tokens still cost 0.71-0.95x (Obs 2). Policies with matching resolve rates solve different tasks (Obs 4). Policy rankings reverse across models, with one policy gaining 7.0 points on Qwen and losing 17.0 on Devstral (Observation 5). We would welcome pointers to specific studies the reviewer has in mind, and we will add a direct comparison with them.
R3W2: Actionable guidance. Our study gives the community three things it did not have. The first is a common design space. Primitive, trigger and depth describe every deployed mechanism we surveyed, so new methods can be compared on the same terms. The second is an evaluation standard. We show that token savings do not predict cost or latency, and that equal resolve rates can hide different solved tasks. A method evaluated on tokens and aggregate success alone can therefore look better than it is. The third is a reference set of 65 policies across three models and two benchmarks, which any new method can be measured against. The study also gives guidance that can be used today, which previous work do not establish. Keeping recent history verbatim and clearing tool outputs before summarizing each improves SWE-bench resolve rate by 6.7 to 9.7 points on all three models, with every 95% interval excluding zero (Figure 5b). It also identifies open problems for future work, such as why policy rankings reverse across models and how to compress without invalidating the prompt cache.
R3W3: Evaluation criteria. We would be glad to address this more precisely if the reviewer could point to the specific comparisons they found unclear. In the meantime, we note that the paper already supports matched comparisons. Figure 2 plots each policy’s success against its token use, latency and cost, so policies can be read at equal token use or equal success. We report these metrics separately because the acceptable trade depends on the deployment, and a single score would hide it. Where an explicit rule is wanted, dominance provides one, and tool-result clearing is never dominated across our six model and benchmark settings.