- Rating
- 2(+1)
- Confidence
- 4(+1)
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies context compression for long-horizon LLM agents. It proposes that redundant messages are close in embedding space, but there still exist some important evidence that can not be deleted. The paper thus designs a training-free compressor GEM, which designs some heuristic metrics and rules to select and protect these evidence. Experiments show that GEM can save ~20% token usage without losing performance.
Critical strengths
- On the 260-task WorkBuddyBench, GEM reports a 21.4% token usage reduction, with performance keeping at ~70%, demonstrating that the proposed method may be effective in specific tasks.
Critical weaknesses
- The geometry property of redundant messages of LLM agents is not clearly verified. The refered related work (word embedding, bert, simcse) is much different with modern LLM agents and cannot serve as evidence. Fig.1 shows the low-rank structure of the aggregated trajectory embeddings, however, the details of these trajectories and the inference configurations are not disclosed.
- GEM relies on heuristic rules like recency, similarity and tool-calling patterns. However, the justification and necessity of these rules are not verified. The construction of state query, method of error detection, ranking of candidates are also not clearly demonstrated.
- The main experiments are conducted on a single WorkBuddyBench with only 260 tasks, and the other ablations use only 20 or 40 tasks. Such small evaluation scale cannot support the soundness and generality of the proposed method.
Questions for authors
- Is there any clear evidence proving that the low-rank structure of the aggregated trajectory embedding indicates the redundancy of the history messages? How are the trajectories in fig.1 sampled, are these trajectories representative?
- Please provide the implementation details of the evidence-protection rules, and what is their contribution to the proposed method?
- What's the performance of GEM on other common agent benchmarks (e.g. tau-bench, swe-bench)?
- What's the total cost of GEM? Though the token usage is reduced, the reconstructed context cannot hit prefix cache and enjoy discounts for cached tokens. Hence, the final cost may not be cheaper than other baselines.
14 comments
Response to Weakness 2: Justification and contribution of evidence-protection rules
Thank you for your careful examination of the evidence-protection rules. Your request for both implementation details and component-level evidence helps us make the method more transparent.
The rules preserve complementary execution cues: immediate working context, task relevance, artifact changes, recent failures, and information sources. We supplement the implementation description with an online component ablation on 40 fixed tasks, with 10 tasks from each domain.
Each variant starts from the same initial environment for each task. The acting model, tools, task time limits, evaluator, and stopping rules are held fixed.
Online component ablation. Combined includes model input, model output, and embedding tokens. Values are per task.
| Method | N | Reward, mean ± SD | Reward variance | Combined, mean ± SD |
|---|---|---|---|---|
| Full GEM | 40 | 0.7073 ± 0.2246 | 0.0504 | 2.50M ± 2.10M |
| Geometry-only | 40 | 0.6468 ± 0.2191 | 0.0480 | 2.85M ± 2.73M |
| w/o Recent | 40 | 0.6299 ± 0.2081 | 0.0433 | 2.80M ± 2.21M |
| w/o Goal | 40 | 0.6513 ± 0.1938 | 0.0376 | 2.70M ± 1.94M |
| w/o State | 40 | 0.6732 ± 0.2281 | 0.0520 | 2.75M ± 2.56M |
| w/o Error | 40 | 0.6587 ± 0.2097 | 0.0440 | 2.62M ± 2.44M |
| w/o Source | 40 | 0.6724 ± 0.2256 | 0.0509 | 2.58M ± 2.44M |
Paired comparisons. Reward differences are Full GEM minus the corresponding variant; positive token savings indicate lower Combined for Full GEM.
| Variant | Reward difference | Paired-difference SD | 95% CI | Token savings | 95% CI |
|---|---|---|---|---|---|
| Geometry-only | +0.0605 | 0.2909 | [−0.0248, 0.1470] | 12.28% | [−18.72%, 33.64%] |
| w/o Recent | +0.0775 | 0.2939 | [−0.0080, 0.1588] | 10.71% | [−9.62%, 28.11%] |
| w/o Goal | +0.0561 | 0.2831 | [−0.0295, 0.1407] | 7.41% | [−18.20%, 28.20%] |
| w/o State | +0.0341 | 0.2853 | [−0.0460, 0.1134] | 9.09% | [−8.65%, 23.23%] |
| w/o Error | +0.0486 | 0.2876 | [−0.0385, 0.1342] | 4.58% | [−23.74%, 25.70%] |
| w/o Source | +0.0349 | 0.3124 | [−0.0544, 0.1186] | 3.10% | [−24.92%, 23.57%] |
Full GEM has higher Reward point estimates than all six variants, with differences of approximately 0.034–0.077. The paired intervals span zero, reflecting substantial task-level variability in this 40-task experiment. The token-saving intervals also span zero, so we report the estimates together with their uncertainty.
Our response to Question 2 provides the block definition, state-query schema, error detection, candidate ordering, protection-membership audit, offline component analysis, and a comparison at identical retained block counts. These analyses separately examine implementation, representation-level evidence retention, and downstream execution.
Response to Weakness 3: Evaluation scale and coverage across benchmarks
Thank you for highlighting the evaluation scope and the evidence needed to assess generality. Your comment encouraged us to extend the experiments beyond a single benchmark and agent harness.
We added evaluations on three benchmarks:
- τ-bench: all 165 tasks, comprising 50 airline and 115 retail tasks.
- Terminal-Bench 2.0: 89 tasks.
- SWE-bench Verified: a fixed subset of 100 tasks.
These experiments add 354 tasks covering interactive tool use, terminal operations, and software repair. Within each benchmark, Base and GEM use the same acting model and inference parameters, task set, tools, environment, and stopping rules; their difference is the context-management mechanism.
We also supplement the Full260 evaluation with task-paired uncertainty analysis. The overall mean Reward difference, GEM minus Base, is +0.003069, with a paired-bootstrap 95% CI of [−0.01971, +0.02568]. This describes uncertainty in the overall result alongside the domain-level differences.
The 20- and 40-task experiments address specific mechanisms through controlled comparisons. We report their task counts and uncertainty separately from the full benchmark evaluations. Our response to Question 3 provides the configurations and complete results for the three added benchmarks.
We will revise the conclusions to reflect the observed quality–resource trade-offs in these evaluated settings and the variation across task types.
Response to Weakness 1: Geometric evidence and trajectory documentation
Dear Reviewer WHDb,
We sincerely thank you for the time and care you have devoted to reviewing our work. We also appreciate your recognition of the token reduction and comparable overall performance on WorkBuddyBench. Your detailed questions about geometric evidence, implementation, evaluation coverage, and system costs have helped us identify where the paper needs clearer explanations and stronger empirical support. We respond below to your three weaknesses and four questions in order.
Weakness 1: Evidence for the geometric properties of modern agent histories.
Thank you for emphasizing the need for direct evidence from modern agent trajectories and reproducible descriptions of the data and inference settings.
We agree that studies of word embeddings, BERT, and SimCSE provide background rather than direct evidence about modern agent histories. Our empirical claim is grounded in the agent trajectories analyzed in this work.
We distinguish two properties:
Low-rank structure characterizes the first property; assessing the second requires evidence about subsequent actions and task outcomes. This distinction motivates GEM's evidence protection before geometric completion.
To strengthen the empirical analysis, we provide:
The numerical results and sampling details are provided in our response to Question 1. We will revise the presentation to make the distinction between representational structure and execution relevance explicit.