- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper argues that just measuring task completion is lossy when it comes to compression costs. They propose a measuring protocol to capture these costs with higher fidelity. Instead of just measuring task success, they also measure execution and retrieval tool calls. The idea being that the overall cost consists of execution on the one hand, but also increasing retrieval cost under heavy compression. The choices for how this is set up in detail and why are difficult to follow.
Critical strengths
The idea is sound, and the idea behind the measurement is well-motivated. Compression can incur costs beyond task success, and measuring that sub-failure cost could be useful to guide model context design.
Critical weaknesses
- Writing clarity is very low to incomprehensible, and the text often contradicts itself. One example is the opening of the method section (l. 137), which seems to be a figure description and not an actual opening to the method section.
- The main contribution is a measurement protocol, and its design is not well-motivated. For example, the text in Figure 4 says, “Termination is governed by the fixed 24-turn horizon,” but why were 24 turns chosen? Another example: the “summary” operator does not actually summarize the preceding context but strips the reasoning. Overall, it's entirely unclear why these choices were made and what conclusions, if any, can be drawn from this.
- The novelty statement is contradicted by cited work. L. 91 states, "To our knowledge, none defines a black-box, model-agnostic measure of how much interaction budget an agent spends to reacquire state that compression removed,” but Min et al. cited in l. 50 do that, as (by the work's own statement), they “measure blocked actions and repeated exploration after compression,” which falls in the category of the aforementioned measures.
Questions for authors
Can you please explain plainly why the measurement protocol is set up exactly the way it was and why this is an interesting measurement? Do these insights generalize, e.g., to (significantly) more than 24 turns?
The abstract states GPT-5.6 Sol completion is "unchanged," but the paper also states, “Completion degrades as reacquisition consumes the horizon,” and l. 723 states, “completion fell from 89% to 77%.” How do these statements add up?
How was the p-value of "<.001" in Fig. 6 for tau-bench estimated? Appendix A.11 states that ten tasks have been used, which would not allow obtaining such a low value. If the re-runs of the same tasks are considered "independent" that would be rather questionable. L. 408 states "p = 0.002", which seems correct for ten tasks, but in contradiction with the figure.
What do the columns "Wilcoxon p" and "exact perm p" in Table 12 measure?