- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how to compress long tool outputs in LLM agents. It argues that keeping many relevant facts ("fact coverage") is different from telling the agent how a result confirms or overturns what it expected ("evidence updating"). The proposed method, EDO, has the main agent write a short expected observation in the same message as each tool call. After execution, a small model (Qwen3.5-4B) compares the raw output with and writes a natural-language residual organized around support, contradiction, extension and unresolved claims. The pair replaces the raw output in later context. Short outputs pass through unchanged, and file reads and TUI outputs get deterministic compaction. On six coding and terminal benchmarks (Qwen3.7-flash solver, three runs), EDO keeps 98.97% of Raw's macro score. It uses 59.06% of Raw's main-agent tokens and 60.79% of its estimated cost. On ProgramBench it beats an expectation-conditioned summary by 4.10 points [1.44, 6.83], and the gain holds with a second solver and three prompt paraphrases. Offline diagnostics on 600 observations show higher correction recall (94.55% vs 70.00%) at similar overall fact recall. An appendix gives a short argument that high fact recall need not preserve decision-relevant distinctions.
Critical strengths
- The framing/writing is clear and the method is simple and practical. Separating fact coverage from evidence updating is a useful lens. EDO is a drop-in change that needs no training, and the expectation costs only about <1k solver tokens per task. On ProgramBench, main-agent input drops significantly per task and latency drops to 44 s, at a cost of 2.8 points against Raw. This matters because it shows a real efficiency gain from a method that is easy to adopt.
- The ablation and experiment is careful and transparent. Costs are cache-aware and include auxiliary calls. The imputed auxiliary price comes with a sensitivity analysis: EDO stays 52% below Raw even when the 4B model is priced at solver rates. A step-matched prefix comparison separates token saving from shorter trajectories. This makes the efficiency claims credible even where the accuracy claims are not.
Critical weaknesses
My concern is mainly around the experimental results:
-
The evidence that the residual itself matters rests on a weak baseline. The main controlled comparison is against Expected + Conditional Summary. On ProgramBench, however, this baseline scores below both summaries that never see the expectation: about 22.5% against 24–26% (Tables 2 and 13). Adding the expectation actually hurts the summarizer, so the matched control is the weakest summary in the paper. Under the default setting, the differences from simpler alternatives are within noise. Those alternatives are Generic Summary, Goal Hint, and Recent-Observation Masking, which needs no auxiliary model. The ablation without the expected observation does not help either. It falls far below every summary, which shows only that the residual prompt breaks without an expectation, not what the residual adds. The 20% budget results do not settle the question. There, more than half of EDO's representations and over 40% of the matched summary's are hard-truncated (Table 12), so that setting tests truncation and ordering rather than the representation. The paper's central claim is that expectation-relative organization, not fact coverage, is what makes compressed feedback useful, and the current evidence does not support it.
-
The general baselines look misconfigured outside ProgramBench, and the headline numbers depend on them. On TerminalBench, all five general baselines score roughly half of Raw, and all five cost more than Raw (Tables 2–3). QuixBugs and DevEval show similar negative savings. This is hard to reconcile with Table 39, where the controls on the same benchmark come close to Raw. That includes masking, which uses plain head–tail truncation and no model. AgentDiet only rewrites older steps, yet it loses about 30 points. Either the Table 2 baselines were run under a different or broken setup (for example, the default Head-Tail budget is never given), or the expectation prompt itself changes solver behavior. On this benchmark, Expected + Conditional Summary scores almost 20 points above Task-Conditioned Summary. In both cases, the claims of "best on five of six" and "largest cost saving on five" do not compare like with like. Two related issues:
- On TerminalBench, the comparison against Goal Hint and Masking, the two strongest controls, is not reported. There is also no Expected + Raw run to separate the effect of the prompt from the effect of compression.
- The macro average gives the 20-task CompileBench the same weight as SWE-bench Lite, even though a gap there amounts to a few runs out of 60. The method is below Raw on four of six benchmarks, and without CompileBench and TerminalBench the ratio drops to about 94%.
The cross-benchmark efficiency–quality trade-off is the paper's main practical claim, so this weakens it considerably.
Questions for authors
- The expectation-conditioned summary is explicitly told to correct contradicted expectations, yet it scores under 25% on "contradicted expectations corrected." What wording does this metric require to count a correction? Does it mostly reward the residual's "expected X, observed Y" phrasing? The diagnostic pool also comes only from EDO trajectories. Could that bias the comparison?
- How were the Table 2 baselines configured on TerminalBench, QuixBugs and DevEval? In particular, what Head-Tail budget and passthrough thresholds were used, and are the wrappers the same as for Table 39? Why do the summaries and AgentDiet lose about 30 points on TerminalBench while masking loses almost nothing?
- Almost every count in the diagnostic table of Appendix C.1 is a multiple of 20. Are these raw annotation counts over the 600 observations, or were they rescaled or estimated? How were the fact units built? Was the single annotator blind to which method produced each output? The units were first proposed by Qwen3.5-4B, which is the same model as the compressor.
7 comments
W2 and Q2. Baseline configurations, Table 2 versus Table 39, prompt effects, and benchmark weighting
Thank you for raising this concern. We re-audited the baseline implementations and configurations. Within each benchmark, Table 2 and Table 39 use the same task sets, environment/tool wrappers, solver settings, and evaluators; the observed differences arise from what the compression policies retain. Table 2 reports Raw, Generic Summary, Task-Conditioned Summary, LLMLingua-2, Head–Tail, AgentDiet, and EDO. Table 39 adds Expected + Conditional Summary, Goal Hint, and Masking on TerminalBench (and the matched summary on DevEval). Crucially, EDO and Expected + Conditional Summary use the same expectation-generation prompt, observable input fields, passthrough/deterministic/model-routing decisions, and 1,024-token feedback limit when given the same raw observation.
The principal configuration details are:
- Head–Tail: 1,024 tokens, split equally between the beginning and end. LLMLingua-2: 90% target retention in the default Table 2 experiment (the 20%–80% sweep is separate). Model-generated feedback is capped at 1,024 tokens.
- TerminalBench-style routing: empty observations remain empty; outputs shorter than 900 characters pass through; recognized file reads are deterministically compacted up to 4,500 characters; ANSI/TUI observations retain at most 20 lines and 12 visible examples. ProgramBench/mini-swe-agent summary paths use 1,800/2,200-character thresholds depending on conditioning.
- AgentDiet: attempts to rewrite the step two steps behind the current step, with one-step prefix context; it triggers only above 500 tokens and accepts a replacement only if it saves more than 500 tokens. Masking: retains the current observation (3:1 head–tail truncation above 1,024 tokens) and masks older observations.
- Common execution: Qwen3.7-flash solver, Qwen3.5-4B for model-based compression, temperature 0, 30-step limit, three runs, common task/evaluation conditions.
These policies preserve different information. In terminal tasks, exact commands, paths, errors, and exit status can determine the next action; task-focused summarization and trajectory rewriting may omit such details, while a current-observation-preserving policy can retain them. This is a plausible explanation for the large observed differences, not an experimentally isolated causal attribution. In particular, Table 39's additional controls are not simply reruns of Table 2's generic summarizers under an interchangeable instruction.
To separate the effect of eliciting an expectation from the effect of compressing feedback, we added Expected + Raw on TerminalBench: the solver produces the same pre-tool expectation but receives the raw tool output, without an auxiliary compressor.
| TerminalBench method | Success (3 × 86 task-runs) |
|---|---|
| Raw | 52.33% |
| Expected + Conditional Summary (Table 39) | 119/258 (46.12%) |
| Goal Hint (Table 39) | 132/258 (51.16%) |
| Masking (Table 39) | 129/258 (50.00%) |
| Expected + Raw (new) | 137/258 (53.10%) |
| EDO | 141/258 (54.65%) |
Expected + Raw reaches 53.10%, suggesting that the expectation prompt itself yields only a marginal improvement. EDO preserves this benefit while compressing tool feedback and achieves the highest observed success rate, 54.65%. The table also makes the stronger, previously omitted TerminalBench controls explicit.
Task-count-weighted aggregation. To address the equal-weight macro-average concern, we additionally weight each benchmark by its task count:
| Method | Weighted task score | Relative to Raw | Weighted token saving | Weighted cost saving |
|---|---|---|---|---|
| Raw | 44.39% | 100.00% | 0.00% | 0.00% |
| Generic Summary | 37.26% | 83.93% | 21.70% | 19.11% |
| Task-Conditioned Summary | 36.67% | 82.61% | 20.74% | 18.42% |
| LLMLingua-2 | 36.40% | 81.99% | 6.39% | 3.60% |
| Head–Tail | 33.92% | 76.41% | 21.86% | 18.77% |
| AgentDiet | 35.68% | 80.39% | 27.73% | 20.99% |
| EDO | 42.84% | 96.51% | 40.80% | 40.97% |
EDO retains 96.51% of Raw's task-count-weighted score while achieving the largest weighted token and cost savings among the compared compression methods.
Q1. What counts as a corrected expectation, and is the diagnostic biased toward EDO trajectories?
Thank you for asking us to clarify the criterion. No fixed wording is required. The metric judges whether the compressed feedback semantically preserves the correcting observed fact; it does not require repeating the original expectation or using an ``expected X, observed Y'' template. The paper reports two different correction metrics, with different units:
- Correction recall is measured per corrective fact in the raw observation: the compressed feedback receives credit if it preserves that observed fact, including semantically equivalent wording. In the original fixed-input ProgramBench diagnostic aggregated over 20 generation seeds, the matched summary retains 1,540/2,200 (70.00%), and EDO retains 2,080/2,200 (94.55%).
- Contradicted expectations corrected is measured per checkable expectation refuted by the raw observation. A method receives credit if its feedback directly and unambiguously conveys the contradictory observed fact. An unsupported prediction is not counted as refuted merely because the output did not verify it. We will revise Appendix C.1 to make clear that this metric uses semantic matching rather than a fixed verbal template.
The original diagnostic drew its fixed inputs from EDO trajectories. To test source-trajectory dependence, we independently sampled 600 observations from Expected + Conditional Summary trajectories, then evaluated both compressors against the same fixed observations and shared reference units within each source:
| Source trajectories | Expected + Conditional Summary | EDO |
|---|---|---|
| EDO | 135/571 (23.64%) | 536/571 (93.87%) |
| Expected + Conditional Summary | 234/656 (35.67%) | 603/656 (91.92%) |
EDO's advantage persists when the states are drawn from the summary method's own trajectories, reducing the concern that the difference arises solely from selecting EDO-visited states.
W1. Does the residual representation matter beyond a weak matched summary?
Thank you for this careful assessment. The matched Expected + Conditional Summary is the key control for identifying the effect of the feedback representation: it holds fixed the pre-execution expectation, input fields, auxiliary model, routing, and 1,024-token output cap, while changing whether the output is organized by task relevance or by the expectation–observation evidence update. It also explicitly instructs the summarizer to correct contradicted expectations. On ProgramBench, this yields 26.59% for EDO versus 22.49% for the matched summary, a paired +4.10 points [1.44, 6.83]. The other summary baselines evaluate the full method's quality--cost trade-off, whereas this matched comparison isolates residual organization.
The matched experiment establishes the effect of feedback organization, while comparisons with Generic Summary, Goal Hint, and Masking evaluate the full method's quality--cost trade-off. Two further observations identify what the matched comparison measures:
At a 20% retained-representation budget, EDO also scores 15.27%, compared with Goal Hint 10.65%, Masking 8.21%, and the matched summary 2.28%. We interpret this as an aggressive-compression result encompassing evidence selection, ordering, and hard truncation, rather than a pure test of residual semantics alone.