- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes four checks for assessing tool-observation compression opportunities: structural support, reuse-weighted cost, behavioral influence, and execution integrity. Audits on BFCL and τ2-Bench show that local token savings and state redundancy need not translate into broadly applicable, usable, or cost-effective compression. The contribution is a measurement and experiment-selection workflow rather than a deployable compressor.
Critical strengths
- The paper addresses a practical question: whether a compression direction has sufficient applicability and potential savings to justify further development.
- The audits distinguish structural redundancy from decoder access and net savings, with transparent reporting of negative results and limitations.
Critical weaknesses
- The four audits are reasonable experimental principles, but their added value over conventional evaluation is insufficiently demonstrated. The cases primarily expose limitations of specific candidates, rather than establish that the framework reliably identifies promising compression directions.
- The paper lacks an overview diagram or running example. The experiments use different interventions, populations, and endpoints without clearly explaining how they jointly validate the framework, making both the technical approach and the evidence chain difficult to follow.
Questions for authors
- Why does expansion of the reobservation pilot require sufficiently many action or outcome changes? Compression typically aims to reduce cost while preserving task performance. What subsequent hypothesis is this behavioral-change gate intended to screen?
- Why must the reconstructible and epistemic-only partitions each account for at least 10% of JSON-object bytes, with at least 20 mixed calls? Are these requirements motivated by a particular candidate design, and would alternative reasonable thresholds change the stop/go decisions?
- In the reported cases, which experimental decisions require the complete four-part audit rather than simpler applicability and cost checks? Can the retained evidence demonstrate a specific mistaken decision that the complete framework prevents?
2 comments
The paper is still difficult to follow. A single example that walks through the four checks and shows how they lead to a decision would help.
Thank you for asking us to demonstrate the practical value of the audit and clarify its decision rules.
Across 848 archived histories, structural checks miss all injected same-type referent changes, whereas trusted-binding checks detect all 133 Qwen cases and all 128 Llama cases. This provides a concrete failure that structural validity alone does not catch.
Our new 160-family experiment also distinguishes token reduction from preserved decisions: deleting the latest observation reduces next-step execution equivalence from 94/160 to 73/160 for Qwen and from 82/160 to 68/160 for Llama.
We evaluated 150 combinations of partition and mixed-call thresholds and will report the resulting decision changes. These thresholds screen the specific mixed-partition candidate. The reobservation gate instead screens sensitivity to redundant observations; it is not an acceptance requirement for behavior-preserving compression.
We will add an overview diagram and a running example connecting applicability, decoder access, fully charged cost, and behavioral or integrity checks. The new experiments show why a candidate can save tokens or pass structural checks yet still require additional evaluation.