acceptodds
Under review as a conference paper at ICLR 2027

What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics

Abstract

Task completion is the standard metric for evaluating context compression, yet it is an incomplete measure: compression can substantially increase an agent’s interaction cost — the reacquisition of state it dropped — without any detected change in completion. We frame completion-only evaluation as a lossy projection of the full evaluation outcome and introduce a controlled runtime protocol that measures this reacquisition cost in a bounded-horizon agent. Across three models and two task regimes, retrieval tool calls increase in every one of six model–regime comparisons, five of them after Holm correction, while execution calls change little. At the pre-specified 5× point no completion change is detected in any cell, whereas the retrieval signal already appears: on GPT-5.6 Sol completion is unchanged ([−5, +6] pp; MDE 8.4 pp) while retrieval rises by 24.2 calls. At more aggressive compression, completion degrades as reacquisition consumes the horizon. Oracle restoration and retention interventions probe the mechanism: restoring the queryable dropped state removes most of the added tool cost, while replacing D content with fabricated irrelevant state raises retrieval without a detected completion change. Random retention does not differ detectably from an offline hindsight oracle, suggesting state validity and presence matter more than fine-grained selection in this regime. The effect is not universal: fact-preserving extractive and abstractive operators avoid the surge, while the dropping operator produces it, and the surge reappears in τ-bench; the ALFWorld probe dropped no context and is uninformative about compression. Context compression can therefore incur hidden interaction cost when execution-relevant state must be reacquired, and completion alone can miss that cost.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, context compression, agent evaluation, tool use, long-horizon agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KosFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that just measuring task completion is lossy when it comes to compression costs. They propose a measuring protocol to capture these costs with higher fidelity. Instead of just measuring task success, they also measure execution and retrieval tool calls. The idea being that the overall cost consists of execution on the one hand, but also increasing retrieval cost under heavy compression. The choices for how this is set up in detail and why are difficult to follow.

Critical strengths

The idea is sound, and the idea behind the measurement is well-motivated. Compression can incur costs beyond task success, and measuring that sub-failure cost could be useful to guide model context design.

Critical weaknesses
  • Writing clarity is very low to incomprehensible, and the text often contradicts itself. One example is the opening of the method section (l. 137), which seems to be a figure description and not an actual opening to the method section.
  • The main contribution is a measurement protocol, and its design is not well-motivated. For example, the text in Figure 4 says, “Termination is governed by the fixed 24-turn horizon,” but why were 24 turns chosen? Another example: the “summary” operator does not actually summarize the preceding context but strips the reasoning. Overall, it's entirely unclear why these choices were made and what conclusions, if any, can be drawn from this.
  • The novelty statement is contradicted by cited work. L. 91 states, "To our knowledge, none defines a black-box, model-agnostic measure of how much interaction budget an agent spends to reacquire state that compression removed,” but Min et al. cited in l. 50 do that, as (by the work's own statement), they “measure blocked actions and repeated exploration after compression,” which falls in the category of the aforementioned measures.
Questions for authors

Can you please explain plainly why the measurement protocol is set up exactly the way it was and why this is an interesting measurement? Do these insights generalize, e.g., to (significantly) more than 24 turns?

The abstract states GPT-5.6 Sol completion is "unchanged," but the paper also states, “Completion degrades as reacquisition consumes the horizon,” and l. 723 states, “completion fell from 89% to 77%.” How do these statements add up?

How was the p-value of "<.001" in Fig. 6 for tau-bench estimated? Appendix A.11 states that ten tasks have been used, which would not allow obtaining such a low value. If the re-runs of the same tasks are considered "independent" that would be rather questionable. L. 408 states "p = 0.002", which seems correct for ten tasks, but in contradiction with the figure.

What do the columns "Wilcoxon p" and "exact perm p" in Table 12 measure?

Reviewer_XUneOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the hidden interaction costs of context compression in LLM agents. The authors argue that task completion alone cannot capture the additional retrieval calls caused by discarded context. They introduce a controlled evaluation protocol that separates retrieval and execution calls, and conduct experiments across multiple models, compression settings, and agent environments.

Critical strengths
  1. The paper raises a practically relevant issue: task completion alone may overlook additional interaction costs introduced by context compression.
  2. The experiments cover multiple language models and include several controlled interventions to investigate the observed behavior. The authors provide detailed statistical analyses and acknowledge several limitations of their experimental setup.
Critical weaknesses
  1. The main claim that completion can remain unchanged while retrieval costs increase is intuitive. The non-identifiability formulation in Section 3.1 largely follows from projecting ((Q,C_R,C_E)) onto (Q), without offering much additional theoretical insight.
  2. Most results come from the synthetic IRBench with only 10 tasks and explicit state-query tools. In Figure 6, the effect is smaller in τ-bench, while the ALFWorld experiment is uninformative because compression was never triggered.
  3. The paper defines interaction cost through tool-call counts, but does not measure inference latency, token savings, or monetary cost. It is therefore unclear whether the additional retrieval calls outweigh the benefits of compression.
  4. Figure 6 shows a large retrieval increase under sliding-window deletion but almost none under fact-preserving summaries. Since these summaries are explicitly designed to retain relevant state, it remains unclear whether the findings extend to realistic, lossy summarization methods.
  5. Several experiments reinforce the same observation without leading to substantially deeper conclusions. For example, the Random vs. Hindsight comparison in Figure 5 shows no detectable difference, but the authors acknowledge that the tested budgets already retain much of the relevant state. The practical implications of this result remain unclear.
Questions for authors
  1. What is the main methodological contribution beyond the intuitive observation that removing relevant context can increase retrieval calls?
  2. How would the conclusions change when evaluating realistic, potentially lossy context summarization methods rather than sliding-window deletion?
  3. Does the additional retrieval overhead actually outweigh the inference-time and token savings achieved through context compression?
  4. Can the authors provide stronger evidence that the findings generalize to realistic long-horizon agent tasks?
Reviewer_ej4sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discussed a hidden issue in evaluating context compression in agent evaluation. Only using task completion as a standard metric to evaluate the context compression is incomplete, as the compression could actually trigger the agent to make more tool calls to get back the information lost during compression. To verify the statement, the author built a controlled test on a deterministic project-planning environment (IRBench) with a fixed 24-turn horizon and split the tool calls into retrieval calls and execution calls based on the tool that was used. Full context with sliding window deletion operator at several compression ratios has benn tested on three models (DeepSeek, Qwen and GPT-5.6 Sol). At the 5x ratio setup, retrieval calls go up in all six model-regime comparisons; five of them increase significantly after Holm correction, while there is no change in completion. Followed experiment also supports the discussion: Giving the dropped state back in queryable form removes most of the extra calls (for example, 72.9 → 35.8 tool calls on DeepSeek, Table 4), replacing the kept state with made-up content, which raises retrieval calls by 57% with no detected change in completion. Outside IRBench, τ-bench shows the same direction (+6.00 retrieval-like calls per run, reward 56% → 46%).

Critical strengths
  1. Strong Causal Controls: The paper experimented with why compression adds retrieval calls, not only whether it does or not. Replacing the kept state with made-up content raises retrieval by about 57% from 20.4 to 31.9 calls, while restoring the queryable dropped state removed most of the added calls. Fact-preserving extractive and abstrative summaries also avoid the increase. Overall, these experiments show that the additional cost comes from losing usable state, not simply from making the context shorter.

  2. A transparent, reusable protocol: The paper provided a protocol that other people will find easy to apply to teach compression methods. The author split the retrieval and execution calls. The 5x operating point and primary tests were fixed before the sweep, and the paper reports minimum detectable effects and exact test results. These setups make other people easy to follow and evidence easy to evaluate.

Critical weaknesses
  1. The completion conclusion is too strong. Section 3.4 and Table 3 use only 10 paired seeds. In High-IR, completion drops from 83% to 72% for DeepSeek and from 75% to 66% for Qwen. These drops are not statistically significant, but that does not prove completion is unchanged.

  2. “Interaction cost” is only a proxy. The paper defines it as the number of tool calls, but does not measure total tokens, latency, or monetary cost. Because compression saves prompt tokens while causing additional retrieval calls, the main results do not establish whether total agent cost actually increases.

  3. Most evidence comes from synthetic IRBench; τ-bench uses a different agent setup, and ALFWorld never triggers compression. Also, separating retrieval and execution by tool identity is clean in IRBench but may be less clear for real tools that do both.

Questions for authors
  1. Could you rerun the 5× Full-versus-Sliding comparison in High-IR for DeepSeek and Qwen in larger number of paired seeds, then test whether completion is equivalent within a pre-set ±5-point margin? Please report the paired confidence intervals as well as the p-values.

  2. On those same paired runs, could you measure the complete cost of each condition: model input and output tokens, compression overhead, wall-clock time and monetary cost? Please report the paired difference with confidence intervals, so we can see whether the saved prompt tokens outweigh the extra retrieval calls.

  3. Could you run Full context and 5× Sliding context on the same fixed set of SWE-bench Verified tasks, using the same model, prompt, tools, decoding settings and step budget? Please report solve rate, repeated file-read or search calls, edit and test calls, total tokens, wall-clock time, monetary cost, and how often compression was actually triggered.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.