- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DRSR (Direct Relational Set-Risk Pruning) compresses the context of long-horizon LLM agents by deciding which sets of past history Blocks can be deleted together, rather than scoring each Block on its own. The authors argue that singleton scores cannot capture set risk because of redundancy interactions and accumulated small harms, and they back this up with measurements showing that the best singleton aggregator reaches only about 59% accuracy at ranking deletion sets and that interaction effects grow with set size. Offline, they build labels from 566 completed trajectories by jointly deleting candidate sets and measuring the change in teacher-forced likelihood of the agent's recorded next output. A small scorer, fed features from a frozen Qwen3.5-9B encoder that relate each Block to the current pre-action state along with pooled representations of the deleted set, the retained set, and pairwise interactions, predicts harm and an unsafe score for every candidate set. Online, DRSR batch-scores all valid subsets of up to three Blocks, removes the one with the largest token saving that passes a risk threshold, and abstains if none does. On WorkBuddyBench with a DeepSeek-V4-Flash acting model, it raises reward from 0.699 to 0.802 while cutting tokens by 21% on 260 tasks, and on a 40-task subset it beats 15 context-management baselines with 36% fewer tokens. Ablations show that retained-context features, pair pooling, and abstention each contribute to the gains.
Critical strengths
- The set-level framing is well-argued. The telescoping decomposition and interaction residual make the argument precise, and Section 5.1 backs it empirically.
- The supervision signal is clean and cheap. Fixing the recorded next output and rescoring it under a deleted context avoids the confound of resampling the agent, needs no human labels, and converts a quantity unavailable online into an offline training target.
- Safety constraints are built into the design. Protocol validity, recent-step protection, a 20% removal cap, and abstention when no set passes the gate. The abstention ablation shows it matters.
Critical weaknesses
- The gains are mostly not attributable to pruning. On Eval40, direct deletion removes about 25K tokens per task, but total consumption falls by about 676K per task. Pruning explains under 4% of the saving. The rest comes from the agent taking different, shorter trajectories, and a 0.10 to 0.12 reward improvement from deleting a few old Blocks is unexplained. Appendix G.3 defines this rollout residual but the paper never reports or analyzes it. Without that decomposition, the central causal claim is unsupported.
- The risk scorer is never evaluated out of sample. Section 5 lists pair accuracy, Spearman correlation, AUROC, and Brier score as the offline metrics, and Appendix G.2 defines them, but no held-out values appear anywhere. The reader cannot tell whether the learned gate actually predicts harm or whether the end-to-end effect comes from the structural constraints alone.
- Baseline configuration is opaque. Fifteen baselines, many reimplemented from 2026 preprints, with no description of hyperparameters, prompts, or budgets. Some results are hard to believe, such as the 5-step sliding window consuming more tokens than the uncompressed agent, which hints at failure loops rather than a fair comparison.
- The rule-based baseline is weak. Structural Pruning removes only short transition narration, so it is not a serious alternative. A stronger non-learned baseline, such as sum-of-singleton harm with the same constraints, would test whether the set-level scorer adds value over the constraints and candidate rule.
Questions for authors
- On Eval40, direct deletion removes about 25K tokens per task while total consumption drops by about 676K. Can you report the rollout residual defined in Eq. 40 per domain, along with mean steps per task and retry or failure counts for Baseline versus DRSR? What mechanism explains a 0.10 reward gain from removing a few old Blocks?
- What are the held-out test-split values for unsafe AUROC, Brier score, pair accuracy, and within-state Spearman correlation? These are listed as the offline metrics but no numbers appear.
9 comments
Response to W2: Held-Out Evaluation of the Risk Scorer
W2: Held-out risk-prediction metrics are missing.
Thank you very much for pointing out this important reporting gap. We added the held-out results, source-level split, metric definitions, and confidence intervals so that the scorer's predictive performance can be assessed directly.
1. Evaluation on the original 566-source dataset
The original offline supervision dataset contains 566 source trajectories, 1,698 decision states, and 24,126 joint-deletion labels, split 80:10:10 by source trajectory.
The frozen test set contains 57 source trajectories, 171 states, and all legal deletion-set labels within this test set (approximately 2.4K). Confidence intervals use bootstrap resampling clustered by source trajectory.
| Metric | DRSR | 95% CI |
|---|---|---|
| Unsafe-event AUROC | 0.8417 | [0.8062, 0.8754] |
| Brier (H>0.02) | 0.1586 | [0.1412, 0.1771] |
| Pairwise ranking accuracy | 0.7791 | [0.7463, 0.8107] |
| Mean within-state, same-cardinality Spearman | 0.4924 | [0.4195, 0.5594] |
Metric definitions:
- AUROC and Brier evaluate the unsafe event H>0.02.
- Pairwise accuracy includes only valid pairs whose true H values differ by at least 0.02.
- Spearman is computed among sets from the same decision state with the same number of deleted Blocks, then averaged over valid groups.
- All confidence intervals use source trajectories as the clustering unit.
2. Comparison with learned singleton aggregation
We compare DRSR and the learned singleton-sum model using the same source split, frozen extractor, online-visible features, and test sets. DRSR predicts joint harm directly; the singleton model predicts individual harms and sums them to obtain set risk.
| Held-out metric | DRSR joint model | Learned singleton sum |
|---|---|---|
| AUROC | 0.8417 | 0.7928 |
| Brier | 0.1586 | 0.1819 |
| Signed-harm MAE | 0.0374 | 0.0387 |
| Pairwise accuracy | 0.7791 | 0.7360 |
| Spearman | 0.4924 | 0.4493 |
These results evaluate the learned scorer directly and compare joint-risk learning with singleton aggregation under the same feature budget.
We sincerely appreciate your careful attention to this omission. The revision will include these held-out results alongside the end-to-end evaluation.
Response to W3: Baseline Configuration and K=5 Token Usage
W3: Baseline configurations are insufficiently documented, and K=5 uses unexpectedly many tokens.
Thank you very much for these reproducibility questions. We compiled code versions, checkpoints, prompts, hyperparameters, acting models, and budgets, and examined the K=5 tool logs.
1. Baseline implementations
| Comparator | Key configuration | Implementation |
|---|---|---|
| Sliding Window K=5/10/20 | Prefix + latest K complete assistant/tool interactions; validate tool pairing | Reimplemented |
| Periodic Summary n=3/5 | Summarize and persist oldest interactions every 3/5 new interactions | Reimplemented |
| PACE | BGE-M3; tmax=256, bmax=128000, recent=2, tau=0.3, encoder_max_length=8192 | Dedicated launcher/service |
| LLMLingua-2 | Official checkpoint; rate=0.33 | Checkpoint adaptation |
| SelfCompact | min_round=3, min_tokens=40000, period=2 | Official-prompt adaptation |
| ACON-Core | threshold=4096, preserve_last_k_turns=1 | Frozen-prompt adaptation |
| Self-GC | compression_threshold=0.30 | Paper-mechanism implementation |
| LRE | budget=2048, preserve_last_k=5, min_breadth=3 | Historical lightweight scorer |
| ACM | Qwen3.5-9B actor; input 131072, output 16384 | Official-code adaptation |
| CoMem | DeepSeek actor + Qwen3-4B memory; retain 2 turns | Some memory tokens recounted with corresponding tokenizer |
| SAM-Cabeza | soft64k/hard128k, page_max30k, up to 5 pages | Mechanism adaptation |
| SWE-Pruner | threshold=0.5, keep_first=false, overlap=50 | Official-checkpoint adaptation |
| Sculptor | First action must select a memory tool; up to 20 context-tool calls, 128k | Zero-shot mechanism |
| Acting-model setting | CBC context | Cumulative turn cap | Task timeout |
|---|---|---|---|
| ACM: Qwen3.5-9B; other principal configurations: DeepSeek-V4-Flash request routing | 128000 | 256 | ×2 |
2. K=5 repeated reads and token usage
In the original Eval40 results, K=5 uses 2.952M tokens/task versus Baseline's 1.887M (+56.44%), with Reward 0.564 versus 0.675.
WorkBuddyBench tasks accumulate constraints, files, tool outputs, and completed operations. With only five recent complete assistant/tool interactions retained, earlier evidence can leave the active context. Repeated reads then add tool calls and reinsert identical content into subsequent requests.
Among four tasks completed in common with other windows, K=5's repeated-action rate is 10.35%. Repeated successful actions are all reads, averaging eight/task; 7.5/task return identical content.
For the 18 K=5 tasks with complete logs:
| Group | N | Model calls/task | Identical-content repeated actions/task | Total tokens/task | Share of tokens |
|---|---|---|---|---|---|
| Exhausted 256-call budget | 4 | 256.00 | 143.00 | 8,751,900 | 58.40% |
| Budget not exhausted | 14 | 51.29 | 13.64 | 1,781,000 | 41.60% |
Four tasks constitute 22.22% of this diagnostic subset but consume 58.40% of its tokens, averaging 4.91 times the other tasks' usage.
3. Log example
Under K=5, canvas-webgl-scene-L4-026 repeatedly reads six files:
| File | Reads |
|---|---|
| mountain-overlook-scene-brief.md | 43 |
| camera-and-lighting-constraints.yaml | 43 |
| scene-graph.json | 43 |
| scene-module-cues.json | 43 |
| scene-build-notes.md | 42 |
| dependency-task-chain.md | 42 |
| Total | 256 |
Only six are first reads; 250 return identical previously read content: 250/256=97.66%. Across all 259 successful tool actions, RAR=250/259=96.53%.
Call-log lines 37–47, 49–59, and 61–71 repeat the same six-file sequence, and the pattern continues. The task reaches 256 model calls, consuming 6,671,700 input and 21,600 output tokens, with native Reward 0.
This case documents how repeated reads can increase total task tokens despite a shorter active context.
4. ACM and CoMem
ACM uses Qwen3.5-9B as its actor, whereas most comparators use DeepSeek-V4-Flash. No actual compression or memory operation was detected for ACM in this run; the reported Reward corresponds to that execution configuration.
CoMem uses a DeepSeek actor, Qwen3-4B memory model, and two retained turns. Earlier constraints, paths, completed operations, and tool outputs may not all be preserved. Its Security subset has low Reward and multiple zero-score tasks. Its outcomes reflect the acting model, memory model, and retention policy together.
Thank you again for helping us improve implementation transparency and connect unusual token results to concrete execution logs.
Response to W1: Attribution of Token Savings and Reward Changes
Dear Reviewer,
We sincerely thank you for the time and care you have devoted to this rigorous and constructive review. We especially appreciate your recognition of the set-level formulation, the fixed-output counterfactual supervision, and the protocol, recent-history protection, and abstention safeguards. Your positive assessment is encouraging.
Your detailed questions about savings attribution, held-out prediction, baseline reproducibility, and stronger controls have helped us identify where the evidence and explanation needed strengthening. We greatly value your expertise and the effort you have invested in improving this work. Below, we address your four weaknesses and two questions in order.
W1: Direct deletion explains less than 4% of token savings; the rollout residual and Reward changes require analysis.
Thank you for highlighting the distinction between direct deletion and total task-level token changes. We analyzed per-task token counts, model-request records, and tool logs.
1. Per-domain rollout residual
In ReAct execution, accumulated history is repeatedly included in subsequent requests. Deleting history can therefore save tokens in later requests as well as the current one. It can also change subsequent outputs, tool interactions, and termination paths.
Using the original Eval40 Table 4 and Appendix I statistics, we apply Eq. (40):
=T_{\mathrm{direct}}+\Delta T_{\mathrm{rollout}}.$$ All values below are per-task averages. | Domain | Baseline (M) | DRSR (M) | Direct deletion (K) | Rollout residual (K) | |---|---:|---:|---:|---:| | Code | 1.438 | 1.216 | 46.6 | 175.4 | | Office | 1.040 | 0.724 | 6.8 | 309.2 | | Security | 3.973 | 1.616 | 37.6 | 2,319.4 | | Web | 1.098 | 1.288 | 9.7 | -199.7 | | Overall | 1.887 | 1.211 | 25.155 | 650.845 | Total savings are 676,000 tokens/task: direct deletion contributes 25,155 (3.72%), and the residual contributes 650,845 (96.28%). The residual is defined as the total-token difference minus direct deletion. It combines changes in repeated history input, request lengths and counts, output lengths, tool interactions, and termination paths. Its positive values in Office and Security and negative value in Web show that subsequent execution costs vary by domain. **2. Model responses, retries, and tool errors** From per-task execution logs, we report main-model responses, exact retries following failures, and tool errors separately. | Domain | Baseline responses/task | DRSR responses/task | Baseline retries/task | DRSR retries/task | Baseline tool errors/task | DRSR tool errors/task | |---|---:|---:|---:|---:|---:|---:| | Code | 39.40 | 32.10 | 1.26 | 0.61 | 2.60 | 1.40 | | Office | 24.80 | 25.90 | 0.42 | 0.48 | 0.90 | 1.10 | | Security | 88.70 | 63.20 | 4.35 | 2.64 | 5.70 | 3.60 | | Web | 46.30 | 48.10 | 1.08 | 1.21 | 1.70 | 1.90 | | Overall | 49.80 | 42.33 | 1.78 | 1.24 | 2.73 | 2.00 | **3. Why a small deletion can affect Reward** Deletion changes the context visible to later decisions. Removing redundant or outdated history may reduce distraction and alter state judgments, tool choices, and the subsequent execution path, in addition to reducing repeated input. In the original domain results, Security has the largest Reward increase (+0.254) and rollout residual. Code and Web improve by 0.127 and 0.123, while Office declines by 0.028. Web improves in Reward despite a negative token residual. We therefore report Reward and each cost/process measure separately; these changes are not uniform across domains. **4. Repeated full-task execution** We additionally ran the original 40 Eval40 tasks with the same environment, files, tool protocol, and acting model, using three independent seeds/task: 120 trajectories/policy. | Policy | Tasks | Trajectories | Mean Reward | |---|---:|---:|---:| | Undeleted Baseline | 40 | 120 | 0.7113 | | DRSR | 40 | 120 | 0.7237 | DRSR has a higher observed mean Reward in this repeated-run batch. These results are reported separately from the original paper's domain results. Thank you again for helping us make the attribution more precise. The revision will distinguish direct savings, the rollout residual, execution behavior, and Reward rather than attributing the full token reduction to immediate deletion.