- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces StateComp, which uses a router to decide which historical interactions to retain in full or summarize based on the current agent state. The motivation is that the need for the original details of past interactions changes as a task progresses. The authors construct LLM-assisted annotations and train a router on representations from a frozen language model to predict whether an interaction can be compressed. On WorkBuddyBench, StateComp marginally improves task performance while reducing agent and summarization tokens relative to the no-compression baseline. The comparison with diverse compression methods in Table 3 shows the highest mean task reward among the evaluated methods, together with lower token usage than the no-compression baseline.
Critical strengths
- The idea of predicting which past interactions can be compressed based on the current agent state is interesting.
Critical weaknesses
-
I have serious concerns about the reliability and generalizability of the supervision described in Section 3. The KEEP/READY labels and earliest safe compression boundaries are derived through an annotation procedure involving an LLM. Although the authors state that they reviewed the procedure and resulting labels, it remains unclear whether these judgments transfer reliably to other long-horizon agent tasks beyond those evaluated. The annotation model is not disclosed, and the paper does not establish how the choice of annotating LLM affects label quality or the performance of the trained router.
-
The empirical evidence is insufficient to establish broader generalizability and practical efficiency. Although the paper evaluates StateComp against no compression on 260 tasks, the comparisons with diverse existing methods in Table 3 are limited to 40 tasks. This restricts the strength of the comparative evidence. The implementation and evaluation settings for these baselines are also insufficiently explained. In addition, although representation-extraction runtime and GPU memory are reported separately, the full deployment cost of the representation model and router is not quantified end to end. The reported token savings therefore do not establish the overall efficiency gain once this computation is included.
Questions for authors
-
Approximately 93.6% of interaction-state pairs in the supervision data are labeled KEEP. How do the KEEP/READY proportions change during online evaluation? Please report these proportions, the fraction of interactions actually summarized, and the corresponding token volumes to clarify where the reported token savings come from.
-
KEEP and READY are used in the Introduction before they are defined. Could the authors define these labels when they first appear?
7 comments
Evaluation, Costs, and Offline Router Results
Weakness 2: Baselines, evaluation scope, and deployment efficiency
Evidence for broader generalizability and practical efficiency is insufficient. The 260-task evaluation compares against no compression, but Table 3's multi-method comparison covers only 40 tasks, with insufficient baseline implementation and evaluation details. Representation-model and Router deployment costs are not quantified end to end, so token savings alone do not establish overall efficiency.
Thank you for identifying these gaps. The revision will distinguish Full260 evaluation, Eval40 controls, and external benchmarks, and document baseline implementation sources, adaptations, evaluation settings, and token accounting.
2.1 Three seeds on fixed Eval40
Each seed uses the same 40 tasks, giving 120 complete runs per method.
| Seed | Base Reward | τ=0.60 Reward | Base tokens M/task | τ=0.60 tokens M/task |
|---|---|---|---|---|
| 17 | 0.6819 | 0.7215 | 1.8286 | 1.0744 |
| 31 | 0.6680 | 0.7085 | 1.9346 | 1.1278 |
| 53 | 0.6711 | 0.7181 | 1.8908 | 1.0864 |
| Mean | 0.6737 | 0.7160 | 1.8847 | 1.0962 |
All three seeds show higher mean Reward and lower Combined Tokens, adding repeatability evidence on fixed tasks. Full260 and the external benchmarks separately provide broader task coverage.
2.2 End-to-end latency including auxiliary computation
Mean complete-task latency in the same Eval40 evaluation:
| Method | E2E sec/task |
|---|---|
| Baseline | 910.22 |
| STATECOMP | 892.13 |
STATECOMP's E2E time includes Agent execution, summarization, local Qwen3.5-9B feature extraction, and Router forward computation. Combined Tokens count Agent and summary Input + Output; local computation is reported through latency and memory.
2.3 Feature extraction and local resources
STATECOMP uses a frozen feature model and lightweight classification head without autoregressively generating READY/KEEP judgments. It incurs no additional generative Router API bill, but uses local GPU compute, memory, and model-residency resources.
Paired measurements from Appendix C.3:
| Metric | Full history | Bounded representation |
|---|---|---|
| Paired feature-extraction time | 76.40 sec | 6.03 sec |
| Mean latency/state | 10.914 sec | 0.861 sec |
| Peak GPU memory | 29.426 GiB | 18.620 GiB |
| Forward workspace | 11.867 GiB | 1.062 GiB |
| Throughput | 0.0916 states/s | 1.1611 states/s |
Bounded representations provide a 12.67× extraction speedup, 36.72% lower peak memory, and 91.05% less forward workspace. This speedup concerns feature extraction; E2E latency is reported separately above.
Local feature-computation times on external benchmarks:
| Benchmark | Valid timing N | Mean 9B CUDA sec/task |
|---|---|---|
| SWE-bench Verified | 100 | 13.2817 |
| τ-bench | 165 | 2.2544 |
| Terminal-Bench 2.0 | 89 | 10.6537 |
CUDA times measure active local computation. Bounded representations, non-autoregressive classification, and execution gates control auxiliary overhead. Thank you for helping us separately report offline annotation cost, online token usage, local computation, and complete-task latency.
Question 1: How do 93.6% KEEP, online READY, and actual compression relate?
Approximately 93.6% of supervision interaction-state pairs are KEEP. How do online KEEP/READY proportions change? Please report these proportions, the fraction of interactions summarized, and corresponding token volumes to explain the savings.
Thank you for requesting clearer accounting. The original supervision is approximately 93.6% KEEP and 6.4% READY. Offline pairs, online interactions, and committed spans use different counting units; we report them separately.
3.1 Independent offline threshold evaluation
The final Router uses Weighted BCE () and frozen Qwen3.5-9B Layer 32 features. Its independent test set contains 110 trajectories and 3,732 interaction-state pairs.
This set deliberately samples trajectories with more READY opportunities to examine READY recognition. Its label distribution does not represent the original 244,526-pair supervision set.
| Threshold | Test pairs N | Precision | Recall | Predicted READY |
|---|---|---|---|---|
| 0.35 | 3,732 | 70.2368% | 79.9401% | 71.3023% |
| 0.40 | 3,732 | 72.7184% | 63.7297% | 54.9035% |
| 0.50 | 3,732 | 80.7420% | 39.0932% | 30.3323% |
| 0.55 | 3,732 | 83.5821% | 28.7425% | 21.5434% |
| 0.60 | 3,732 | 85.4130% | 20.7870% | 15.2465% |
| 0.65 | 3,732 | 87.9177% | 14.6279% | 10.4234% |
| 0.70 | 3,732 | 90.1099% | 10.5218% | 7.3151% |
| 0.75 | 3,732 | 92.7273% | 6.5441% | 4.4212% |
| 0.80 | 3,732 | 95.0617% | 3.2934% | 2.1704% |
The 15.2465% is the offline test-set prediction rate, not the online READY proportion.
Online Compression and Terminology
3.2 Online READY predictions and successful commitments
Online READY steps count distinct historical interactions predicted READY at least once anywhere in a complete trajectory. An interaction predicted READY at multiple checkpoints is counted once.
| Run (τ=0.60) | Tasks N | Mean READY steps/task | Mean committed spans/task |
|---|---|---|---|
| SWE-bench | 100 | 23.80 | 0.86 |
| τ-bench | 165 | 12.59 | 0.5273 |
| Terminal-Bench | 89 | 35.60 | 1.0787 |
| Full260 | 260 | 26.15 | 1.2308 |
READY candidates must also pass contiguous-span, minimum-source-token, and summary-validity gates, so candidate and commitment counts differ. Although τ-bench histories are generally shorter, some contain eligible intervals and are compressed.
In Full260, approximately 35.51% of online READY candidate interactions enter successfully committed compression spans. The denominator is READY candidates, rather than all historical interactions.
3.3 Commitments and token volumes by domain
| Domain | Tasks N | Successful events | Mean events/task | Previously counted content/tool direct reduction | Summary Input + Output |
|---|---|---|---|---|---|
| Code | 80 | 88 | 1.10 | 36,943 | 77,381 |
| Office | 50 | 23 | 0.46 | 23,868 | 43,653 |
| Security | 60 | 118 | 1.97 | 174,299 | 267,966 |
| Web | 70 | 91 | 1.30 | 103,243 | 165,664 |
| Overall | 260 | 320 | 1.2308 | 338,353 | 554,664 |
Compression-event distribution per task:
| Domain | Tasks N | 0 events | 1 event | 2 events | ≥3 events | Total committed spans |
|---|---|---|---|---|---|---|
| Code | 80 | 40 | 20 | 10 | 10 | 88 |
| Office | 50 | 34 | 11 | 3 | 2 | 23 |
| Security | 60 | 20 | 15 | 10 | 15 | 118 |
| Web | 70 | 35 | 15 | 8 | 12 | 91 |
| Overall | 260 | 129 | 61 | 31 | 39 | 320 |
The earlier 338,353-token count incompletely covered separate reasoning fields. Including them gives:
| Supplementary item | Full260 total | Per-task mean |
|---|---|---|
| Previously counted content/tool direct reduction | 338,353 | 1,301.36 |
| Reasoning-field tokens | 20,038,720 | 77,072.00 |
| Complete-history direct removal | 20,377,073 | 78,373.36 |
| Summary Input + Output | 554,664 | 2,133.32 |
Memory replacement also reduces repeated input in subsequent requests, so one-time removal and cumulative complete-task savings are different quantities. The revision will distinguish supervision labels, offline predictions, online candidates, commitments, direct removal, and cumulative usage. Thank you for helping us clarify these relationships.
Question 2: Define KEEP and READY at their first appearance
KEEP and READY appear in the Introduction before being defined. Could you define them at their first appearance?
Thank you for identifying this issue so carefully. The manuscript was written by the authors themselves and was not generated by AI. We are firmly committed to not using AI to directly generate papers. Nevertheless, during writing and proofreading, we overlooked defining these terms at their first appearance, increasing the reader's burden. We sincerely apologize. We will add the definitions immediately upon first use in the Introduction and check terminology and cross-references throughout. Thank you for your careful review and professional guidance in improving the paper's clarity and rigor.
The first-use definitions will be:
- KEEP: At the current checkpoint, the historical interaction still requires its original details and is not yet ready for replacement by a summary.
- READY: At the current checkpoint, the history prefix contains sufficient supporting evidence to replace the interaction's original details with an accurate summary while retaining information needed for subsequent execution.
We will immediately clarify that READY does not mean immediate summarization. Commitment additionally requires a sufficiently long contiguous span, minimum source length, summary validity, and a shorter resulting history.
Thank you sincerely for recognizing the research idea and for your specific guidance on supervision reliability, evaluation scope, system cost, and accounting. Your careful review helped us identify and improve reporting gaps and clarify the evidence and scope of the method. We deeply appreciate the time and care you have devoted to understanding and strengthening this work.
Supervision Reliability and Generalization
Dear Reviewer,
Thank you sincerely for your careful review and recognition of state-conditioned history compression. Your comments helped us refine supervision, evaluation, costs, and compression reporting. We address your two weaknesses and two questions below.
Weakness 1: Reliability and generalizability of LLM supervision
Section 3.3 and Appendix A.2 describe the two-stage process, but model disclosure was insufficient. We apologize and provide the details below.
1.1 Annotation model and objective
GPT-6 Sol annotated the 244,526 interaction-state pairs. Each interaction is assessed in its current state: whether original details remain necessary or existing history supports replacement by an accurate summary.
We will disclose the model's full name, version, and inference settings, release both stages' full prompts, label rules, and boundary-review procedure, and reference the appendix.
1.2 Two-stage annotation
Stage 1: Independent checkpoint annotation. Historical interactions are assessed using task progress, observation complexity, anticipated next-subtask needs, remaining interaction value, and evidence already in the history prefix.
Records include rationales, evidence locations, initial KEEP/READY labels, and dependencies preventing compression. Checkpoints are judged independently without inheriting READY labels.
READY requires evidence already in the current prefix. Later interactions may reveal information loss and invalidate an earlier candidate boundary, but future information cannot establish that earlier compression was safe.
Stage 2: Cross-checkpoint boundary review. Initial judgments and original evidence for each interaction are reviewed jointly to identify the earliest boundary supported by existing history. Insufficient or uncertain evidence yields KEEP, rather than propagating the first READY judgment.
For interaction Si with boundary ti∗:
yi,k=⎩⎨⎧0,1,k<ti∗k≥ti∗
Without a supported boundary, ti∗=∞ and the interaction remains KEEP. READY depends on the current state and available evidence, rather than being a fixed interaction property.
1.3 Annotation cost and cross-model agreement
The 244,526-pair supervision set cost approximately USD 2,000, mainly for long-context inputs and two-stage annotation. This offline construction cost will be reported separately from online system costs.
We compared GPT-6 Sol and DeepSeek-V4.1-Flash annotations on 29,182 interaction-state pairs and 829 historical interactions from 30 trajectories. Pair-level agreement and Cohen’s κ were computed on pooled pairs; per-trajectory agreement was macro-averaged, and boundary agreement aggregated by interaction.
Agreement on the earliest compressible boundary:
These quantify cross-model label and boundary agreement. We will specify metric definitions, full prompts, label conditions, and boundary review to improve transparency and reproducibility.
1.4 Evaluation across task environments
Three external benchmarks cover software repair, interactive tool use, and terminal operations, each with its native execution protocol.
SWE-bench and Terminal-Bench show higher Reward and lower token usage. τ-bench generally has shorter interactions and less replaceable history, showing smaller savings and a quality–usage trade-off.
Annotation comparisons assess cross-model consistency; benchmarks assess cross-task execution. We will distinguish these to clarify how supervision relates to task performance. Thank you for your helpful suggestions.