acceptodds
Under review as a conference paper at ICLR 2027

When Is Tool-Observation Compression Worth Pursuing? Replay Audits Across Agent Interfaces

Abstract

Tool-observation compression is useful only if a transformation applies to the workload, can be decoded from available information, and reduces model-facing cost without damaging behavior. We present a replay audit separating structural support, reuse-weighted mass, paired behavioral leverage and execution integrity. On pinned BFCL tasks, a numeric-array summary saves 41.81% of prompt-plus-completion tokens on eight selected development families but has no qualifying target in 320 held-out reference variants. A typed-subtree state audit gives a different interface comparison: 8.53% of held-out BFCL JSON-object bytes are reconstructible, versus 100% among successful retail and airline reference actions in -Bench. Access to simulator state, however, does not establish decoder access. History-only controls yield little net saving, and neither tokenizer reaches the frozen 5% savings gate in the always-on counterfactual full-template comparison. A supplementary audit of 848 archived histories shows another boundary: structural checks miss same-type referent changes, whereas trusted fingerprints detect the injected changes but can consume the tool-level savings. In a separate 40-position diagnostic, reference encoding reduces exact JSON reconstruction from 80.0% to 37.5% for Qwen and from 52.5% to 5.0% for Llama. The 12-pair natural-task pilot remains inconclusive. Together, these findings show how interface support, decoder access, and cost accounting can invalidate a promising local compression result. The audit supports experiment selection; it does not establish a deployable codec.

Primary area:
datasets and benchmarks
Keywords:
language model agents, tool observations, compression, replay audits, token cost, execution integrity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pZqcOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes four checks for assessing tool-observation compression opportunities: structural support, reuse-weighted cost, behavioral influence, and execution integrity. Audits on BFCL and τ2-Bench show that local token savings and state redundancy need not translate into broadly applicable, usable, or cost-effective compression. The contribution is a measurement and experiment-selection workflow rather than a deployable compressor.

Critical strengths
  • The paper addresses a practical question: whether a compression direction has sufficient applicability and potential savings to justify further development.
  • The audits distinguish structural redundancy from decoder access and net savings, with transparent reporting of negative results and limitations.
Critical weaknesses
  • The four audits are reasonable experimental principles, but their added value over conventional evaluation is insufficiently demonstrated. The cases primarily expose limitations of specific candidates, rather than establish that the framework reliably identifies promising compression directions.
  • The paper lacks an overview diagram or running example. The experiments use different interventions, populations, and endpoints without clearly explaining how they jointly validate the framework, making both the technical approach and the evidence chain difficult to follow.
Questions for authors
  • Why does expansion of the reobservation pilot require sufficiently many action or outcome changes? Compression typically aims to reduce cost while preserving task performance. What subsequent hypothesis is this behavioral-change gate intended to screen?
  • Why must the reconstructible and epistemic-only partitions each account for at least 10% of JSON-object bytes, with at least 20 mixed calls? Are these requirements motivated by a particular candidate design, and would alternative reasonable thresholds change the stop/go decisions?
  • In the reported cases, which experimental decisions require the complete four-part audit rather than simpler applicability and cost checks? Can the retained evidence demonstrate a specific mistaken decision that the complete framework prevents?
2 comments

Thank you for asking us to demonstrate the practical value of the audit and clarify its decision rules.

  1. What additional checks reveal.

Across 848 archived histories, structural checks miss all injected same-type referent changes, whereas trusted-binding checks detect all 133 Qwen cases and all 128 Llama cases. This provides a concrete failure that structural validity alone does not catch.

Our new 160-family experiment also distinguishes token reduction from preserved decisions: deleting the latest observation reduces next-step execution equivalence from 94/160 to 73/160 for Qwen and from 82/160 to 68/160 for Llama.

  1. Thresholds and the behavioral gate.

We evaluated 150 combinations of partition and mixed-call thresholds and will report the resulting decision changes. These thresholds screen the specific mixed-partition candidate. The reobservation gate instead screens sensitivity to redundant observations; it is not an acceptance requirement for behavior-preserving compression.

  1. Clearer evidence chain.

We will add an overview diagram and a running example connecting applicability, decoder access, fully charged cost, and behavioral or integrity checks. The new experiments show why a candidate can save tokens or pass structural checks yet still require additional evaluation.

The paper is still difficult to follow. A single example that walks through the four checks and shows how they lead to a decision would help.

Reviewer_DgYyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies when tool-observation compression is worth pursuing, considering held-out applicability, repeated-context cost, behavior, and execution integrity. Experiments on BFCL and tau-squared-Bench evaluate several mechanisms. The contribution is an evaluation procedure and qualified negative findings, rather than a new compressor.

Critical strengths
  1. I appreciate the held-out applicability checks. The selected numeric-array examples save 41.81% of tokens, yet no qualifying array appears in the held-out reference variants. This clearly limits the result's scope.

  2. The study charges instruction and integrity overhead and restricts references to visible history. It usefully distinguishes machine decoding from model reconstruction; supplementary results retain negative findings under both always-on and on-demand instructions.

Critical weaknesses
  1. The screening evidence is largely retrospective, including the on-demand instruction variant. The 5% savings threshold also determines whether candidates advance. I would appreciate a clearer rationale and sensitivity analysis before using this procedure to guide research choices.

  2. The behavioral evidence is limited to 12 reobservation pairs and 40 reconstruction positions from 17 families. As acknowledged, these do not establish natural-task performance. Please compare the full-reconstruction diagnostic more directly with CoACT's next-action preservation [C26] and LLMLingua's lossy prompt compression [L23]. The information-loss framework in [R26] also helps distinguish preserving a JSON object from preserving decision-relevant information, without equating Bayes-risk bounds with measured LLM accuracy.

I lean toward acceptance for the useful evaluation lessons and careful controls, rather than evidence of deployment savings.

Questions for authors
  1. How do advance/stop decisions change with a savings threshold between 1% and 10%? Which conclusions are independent of that threshold?

  2. Can existing records compare next-action correctness under literal and reference observations? Please explain how the reconstruction test relates to the distinct preservation criteria in [C26, L23, R26]; a conceptual comparison is sufficient here.

[C26] Chen, H., Zhu, Y., Zhang, Y., and Li, J. (2026). CoACT: Action-Preserving Observation Compression for Coding Agents. arXiv:2607.02911. https://arxiv.org/abs/2607.02911.

[L23] Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., and Qiu, L. (2023). LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 13358–13376. https://doi.org/10.18653/v1/2023.emnlp-main.825.

[R26] Ao, R., Gao, S., and Simchi-Levi, D. (2026). On the Reliability Limits of LLM-Based Multi-Agent Planning. arXiv:2603.26993. https://arxiv.org/abs/2603.26993.

2 comments

Thank you for suggesting threshold sensitivity analysis and a direct next-action comparison.

  1. Savings-threshold sensitivity.

We recomputed decisions across thresholds from 1% to 10%. The always-on full-template configurations remain below every threshold in this range. With on-demand instructions, tau-squared-Bench achieves 3.38% savings for Qwen and 2.50% for Llama, identifying precisely where the decision changes. The absence of qualifying numeric arrays in the 320 held-out variants is independent of the savings threshold.

  1. New next-action evidence.

We completed 1,170 test conditions in a paired design across Qwen and Llama. On 21 eligible compression positions, the number of next actions matching the reference execution outcome changes from 12 to 14 for Qwen and remains 10 for Llama under reference encoding. Machine-decoded observations reproduce the literal-condition outputs exactly for both models.

These results directly distinguish complete JSON reconstruction from next-action preservation. We will organize the comparison around three criteria: exact reconstruction, preservation of the next action, and preservation of decision-relevant information.

Thank you for the response. My first question was how the advance/stop decisions depend on the 5% savings gate. The answer settles this: the always-on configurations stay below every gate from 1% to 10%, the on-demand tau-squared-Bench variant would advance only at gates below 3.38% (Qwen) or 2.50% (Llama), and the held-out numeric-array result does not depend on the gate. Please add this to the paper. The on-demand variant is still retrospective and rests on eight tasks, so my view of the screening evidence is unchanged.

The new next-action comparison answers my second question in direction: reference encoding does not reduce agreement with the reference outcome (12 to 14 of 21 for Qwen, 10 to 10 for Llama). With 21 positions, this remains a small diagnostic rather than evidence of behavior preservation. The response does not discuss how the reconstruction test relates to [C26, L23, R26]. The paper cites some of this work but does not compare the criteria, so the positioning against prior work on preserving decision-relevant information still seems incomplete, and a short discussion in the revision would help.

I keep my rating. The response settles the threshold question and adds a useful, if small, next-action check, but it does not change my view that the screening evidence is largely retrospective.

Reviewer_KoMZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors present a replay audit separating structural support, reuse-weighted mass, paired behavioral leverage and execution integrity. On pinned BFCL tasks, a numeric-array summary saves 41.81% of prompt-pluscompletion tokens on eight selected development families but has no qualifying target in 320 held-out reference variants. A typed-subtree state audit gives a different interface comparison: 8.53% of held-out BFCL JSON-object bytes are reconstructible, versus 100% among successful retail and airline reference actions in τ 2-Bench.

Critical strengths

1.It is an important question and solving this can bring real-world impact. 2.The experiments are comprehensive, the authors list the details clearly in the body.

Critical weaknesses

1.The abstract is completely generated by AI, as detected by GPTZero. Some obscure words like “pinned tasks” further proves this. The paper really needs substantial revision on writing, I doubt whether the authors put sincere efforts into the paper.

2.What are these seeds: “20260912”, “20260913”, “20260925”? What does “50-family domain block” mean? What is “auii”? I believe human-written English will not be like this.

Questions for authors
  1. See above, some words are not well defined.
2 comments

Thank you for highlighting the presentation issues.

  1. Authorship and AI assistance.

The manuscript was written by the authors, with AI assistance for language polishing and literature search, as disclosed in the submission. We take responsibility for its content and accuracy and will further revise the abstract and unclear passages.

  1. Clarifying terminology.

“Pinned tasks” means tasks from a fixed benchmark version. The numbers 20260912, 20260913, and 20260925 are reproducibility seeds for resampling and model generation. A “50-family domain block” is a group of 50 task families within a domain, split into 10 development and 40 held-out families. The unclear rendering “auii” refers to tau-squared-Bench. We will replace these expressions with plain language and explicit definitions.

Thank the authors for the clarification.

First, AI coding tools tend to add timestamps to experiments. So if one directly asks Claude Code or Codex to write the paper with little revision, it is not surprising that such timestamps end up in the text. I find it hard to believe that human authors would put timestamps in their own paper.

As far as I know, AI-generated text tends to introduce many terms without defining them. I still think this paper exhibits this pattern. As another example, “held-out” typically refers to samples that are excluded from the training set and reserved for validation or testing. However, on the first page of your paper alone, the term is attached to “reference variants”, “BFCL JSON-object byte”, “tasks”, and “family-held-out-variants”. This is really hard to read, and AI generation is probably the only explanation.

Also, if the authors can convince me that I am wrong, I am more than happy to raise my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.