acceptodds
Under review as a conference paper at ICLR 2027

In-Model Context Compression with Agentic Soft Token for Long-Horizon LLM Agents

Abstract

Long-horizon LLM agents play an important role in solving complex and sustained tasks, with ever-growing interaction contexts. However, these accumulated contexts raise substantial computational costs and may obscure key decision-relevant evidence. Existing methods typically train external compressors or context-management agents to rewrite context as shorter hard-token sequences, but such external selection may not align with the LLM's evolving representation and is necessarily lossy. In this paper, we introduce Agentic Soft-Token Compression (ASTC), a training-free compression method that uses soft tokens within LLM agents for long-horizon tasks. Specifically, ASTC uses the model's native recurrent dynamics to determine already absorbed information, leveraging the commonly used hybrid Gated Delta Network (GDN)–attention architecture. It then carries this information forward as agentic soft tokens while retaining exact values as hard tokens. The resulting compact sequence reduces later-layer computation and yields reusable KV caches, extending the savings across the agent's trajectory. Extensive experiments on long-horizon tasks show that ASTC matches or exceeds the uncompressed agent's accuracy with up to 50.68% fewer end-to-end Attention Pairs, while trained compressors require – more Attention Pairs per successful task. These findings highlight agentic soft tokens as a model-native way to carry absorbed context forward as the task representation evolves. Our code is available at https://anonymous.4open.science/r/agentic-soft-token-8e3b6a/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-horizon LLM agents, agentic context compression, soft tokens

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KkTXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Agentic Soft-Token Compression (ASTC) for hybrid Gated Delta Network–attention models. ASTC uses native recurrent-state signals to select compressible context, protects recognized exact values, and merges selected hidden states into soft tokens. Shorter downstream caches are reused across agent steps.

ASTC uses a reader calibrated from 22 records across 12 Train/Dev tasks. Experiments mainly use Qwen3.5-9B on AppWorld, OfficeBench, and 8-Objective QA, reporting comparable task quality and up to 50.68% fewer analytical attention pairs.

Critical strengths
  1. A coherent training-free mechanism ASTC combines native recurrent signals, exact-value protection, soft-token merging, and persistent caches. This offers a practical direction for reducing agent-context computation without updating model weights.

  2. Useful experimental controls The paper includes selector controls, component ablations, and a fixed-continuation cache-reuse experiment. Removing protection reduces AppWorld accuracy from 33.33% to 20.83%, clearly demonstrating the importance of preserving exact information.

Critical weaknesses
  1. The comparisons do not establish practical efficiency or SOTA superiority

Most hard-token baselines compress full context, while ASTC uses a protected subset, making the comparison difficult to interpret. Training and adaptation details for the learned baselines are also incomplete. Moreover, ACON’s core method uses gradient-free guideline optimization, so labeling it a trained compressor needs clarification. Attention-pair counts include auxiliary calls but do not measure actual latency, memory, or total computation. Relevant internal/cache comparators such as FastKV and KVzip are missing. ACON, FastKV, KVzip

  1. Evidence on harder horizons and the distinctive mechanism remains limited

AppWorld improves by only two successes out of 168; hard deletion is only one success below ASTC. The largest workload reduction comes from OfficeBench, where only repeated, byte-identical interface scaffolds are compressed, making a deduplication baseline important. AppWorld Test-Challenge is not evaluated. Static long-context tests, three-attempt MBPP repair, and near-floor Kimi-Linear results provide limited additional evidence for broad long-horizon robustness.

Questions for authors
  1. Can you provide the learned baselines’ training/adaptation procedures and distinguish ACON guideline optimization from parameter training?

  2. What are the measured end-to-end latency, peak memory, and compression overhead under matched hardware and serving settings?

  3. Does ASTC outperform baselines with identical compression targets, protection rules, and retained lengths? How does OfficeBench compare with cache-compatible deduplication?

  4. How were calibration records, thresholds, and the compression layer selected? Were these fixed before testing? Please provide paired uncertainty estimates and results on AppWorld Test-Challenge or the longest trajectories.

5 comments
AuthorsOpenReview ↗

Response to Review KkTX(1/3)

Thanks very much for your thorough review and insightful questions. We will reply point-by-point below and hope to address your concerns!

q1:Can you provide the learned baselines’ training/adaptation procedures and distinguish ACON guideline optimization from parameter training? & w1:Moreover, ACON’s core method uses gradient-free guideline optimization, so labeling it a trained compressor needs clarification.

Thanks for pointing this out. We have provided these details in our anonymous GitHub repository; all methods follow the settings in their original papers, as described in the appendix of our paper, and the comparisons of ASTC (Agentic Soft Token Compression) with baselines such as LCLM, ACON, and LLMLingua are fully fair. To avoid confusion, we have also provided a more detailed explanation in our revised manuscript as follows: Specifically, ACON requires searching for and selecting guidelines on the training tasks; this offline optimization process incurs actual model calls, token usage, and time costs, and the original ACON paper also states that ACON participates in the training process. We generate initial guidelines and trajectories without compression on Train, then perform two stages of optimization, UT and UTCO, generating, evaluating, and selecting candidates at each stage; the final guideline is fixed at test time. The UTCO optimization model logged 69 calls and 716,220 tokens, and the guideline search and selection process took approximately 19 hours and 25 minutes. We explain ACON's training cost in Figure 2; ASTC does not search for compression guidelines or train compressor parameters, but instead reuses the model's native signals. LCLM uses the publicly available pretrained latent-context/0.6b-4b-LCLM-4x compressor, freezes the 0.6B encoder and existing 1024→2560 adapter, and trains only a newly added 2560→4096 MLP Bridge and LoRA on the Qwen decoder to adapt the representations to the same initial Qwen3.5-9B backbone used by ASTC. LLMLingua v1 uses Qwen3.5-9B for causal token scoring; LLMLingua-2 uses the publicly available pretrained microsoft/llmlingua-2-xlm-roberta-large-meetingbank. For the additionally trained In-model Attention in Figure 2, the inserted Attention–MLP compression module is trained through on-policy distillation.

What are the measured end-to-end latency, peak memory, and compression overhead under matched hardware and serving settings?

Thank you for your suggestion. On NVIDIA A100-SXM4-40GB, we directly apply ASTC to AppWorld and 8-Objective QA, and the results show that ASTC reduces both mean and P95 latency, shortens total queue time, and increases overall queue throughput. We have added these results to our revised manuscript.

DatasetMethodMean latency (s/task) ↓P95 (s/task) ↓token/s/GPU ↑Total queue time (min) ↓Overall queue token/s ↑Peak GPU memory (GiB) ↓Selection overheadPooling overheadTotal compression entry overhead
AppWorldBase73.69210.0824.8737.70136.1021.68———
AppWorldASTC71.00207.4724.4334.43141.0521.465.66413.29246.005
8-Objective QABase182.11339.2726.95——34.72———
8-Objective QAASTC157.62273.7725.54——36.97280.057144.705631.412

Does ASTC outperform baselines with identical compression targets, protection rules, and retained lengths?

Thank you for your question. We added a controlled Hard-token comparison using a frozen LLMLingua-2 scorer. All protected positions are retained, and the final numbers of retained physical positions are strictly identical for the two methods. The results show that ASTC clearly outperforms LLMLingua-2 in both quality and efficiency.

DatasetMethodAcc.% ↑D-KV% ↓Total Attn.% ↓Gen.% ↓
AppWorldASTC33.3381.3780.5694.66
AppWorldLLMLingua-2 (controlled, equal length)26.1977.0598.7890.16
OfficeBenchASTC54.7442.0940.1661.24
OfficeBenchLLMLingua-2 (controlled, equal length)49.4735.5652.7355.79

How does OfficeBench compare with cache-compatible deduplication?

Thank you for your question. We added an exact deduplication comparison on OfficeBench. We retain the initial complete interface, all modified interfaces, and tool observations, and delete only complete copies in newly appended content that are byte-for-byte identical to retained interfaces. ASTC achieves clearly higher quality than the deduplication approach.

MethodAcc.% ↑D-KV% ↓Attn.% ↓Gen.% ↓
Base48.42100.00100.00100.00
ASTC54.7442.0940.1661.24
Cache-compatible exact deduplication50.5341.5239.8072.88
AuthorsOpenReview ↗

Response to Review KkTX(2/3)

How were calibration records, thresholds, and the compression layer selected? Were these fixed before testing? Please provide paired uncertainty estimates and results on AppWorld Test-Challenge or the longest trajectories.

Thank you for raising this question. The original manuscript used empirically set compression parameters, and the layer was determined empirically before the layer comparison in Figure 1, without using that comparison to search for parameters. This time, we conducted a small-scale search over χ and ψ on the dev set and found improvements in both quality and efficiency on OfficeBench and 8-Objective QA. Meanwhile, the parameter choices across the three benchmarks are similar, with some even being identical, demonstrating the algorithm's strong robustness and the potential to reach a higher performance ceiling through further parameter optimization in the future.

DatasetMethodAcc./EM% ↑F1% ↑D-KV% ↓Attn.% ↓Gen.% ↓
OfficeBenchOriginal manuscript47.37—54.5249.3269.23
OfficeBenchRevised54.74—42.0940.1661.24
8-Objective QAOriginal manuscript30.5638.4678.6998.5186.55
8-Objective QARevised30.8139.1857.9674.2982.02

We also added the more challenging AppWorld Test-Challenge. This experiment follows the existing AppWorld settings, and the uncertainty estimates of Attention Pairs are [74.17, 85.12]. We have also provided the longest successful trajectories on AppWorld Test-Challenge in our anonymous GitHub repository.

MethodAcc.% ↑D-KV% ↓Attn.% ↓Gen.% ↓
Base20.86100.00100.00100.00
ASTC19.9087.1379.4395.34

Most hard-token baselines compress full context, while ASTC uses a protected subset, making the comparison difficult to interpret.

Thank you for pointing out this factor. The baselines mainly compress the full context, but we observed that a large portion of an agent's context is generated by tools, and this context both contains compressible redundant information and is challenging to compress; we therefore first applied compression to tool outputs. For a fair comparison, we compared full-context compression in Appendix F of the original manuscript and found comparable results.

DatasetMethodAcc./EM% ↑F1% ↑D-KV% ↓Attn.% ↓Gen.% ↓
AppWorldBase32.14—100.00100.00100.00
AppWorldASTC (Tool)33.33—81.3780.5694.66
AppWorldASTC (Context)32.14—80.5281.8395.73
OfficeBenchBase48.42—100.00100.00100.00
OfficeBenchASTC (Tool)54.74—42.0940.1661.24
OfficeBenchASTC (Context)51.58—37.1140.7658.91
8-Objective QABase28.9137.51100.00100.00100.00
8-Objective QAASTC (Tool)30.8139.1857.9674.2982.02
8-Objective QAASTC (Context)31.3141.4066.8377.7587.40

Relevant internal/cache comparators such as FastKV and KVzip are missing.

Thanks for your question. FastKV and KVzip are works on KV caching and efficient inference, while our research focuses on context compression for long-horizon agents, so FastKV and KVzip do not adequately address the problems in our setting. However, we also conducted additional experiments with FastKV and KVzip following the settings in their original papers. On real-world long-horizon agent tasks such as AppWorld, they clearly underperform ASTC.

MethodTraining-freeAcc.% ↑D-KV% ↓Attn.% ↓Gen.% ↓
Base—32.14100.00100.00100.00
ASTC✓33.3381.3780.5694.66
FastKV✓0.002.53561.2355.40
KVzip✓11.3187.985,976.41145.73

AppWorld improves by only two successes out of 168;

Thank you for your suggestion; research on context compression for long-horizon agent tasks primarily aims to reduce inference costs while maintaining comparable task performance relative to the uncompressed baseline, rather than directly improving the agents' inherent task-solving capabilities. Our work follows this general research objective. Even so, ASTC improves quality on all three benchmarks while providing efficient acceleration. On a real-world and challenging benchmark such as AppWorld, it achieves a +1.19 pp improvement, while Attention Pairs are 80.56% [69.36%, 93.47%] of Base; it also improves quality over the baselines, which require 12.3×–194.9× more Attention Pairs per successful task.

AuthorsOpenReview ↗

Response to Review KkTX(3/3)

hard deletion is only one success below ASTC.

Thank you for raising this question; hard deletion in the ablation study is actually a variant of ASTC that uses ASTC's protection and selection rules but directly deletes redundant, uninformative positions without generating Soft tokens. Therefore, it still benefits from ASTC's internal selection and protection. The direct-deletion method you refer to is the Prompting baseline in Table 1. We have clearly distinguished the two in the revised manuscript.

VariantAcc.% ↑D-KV% ↓Attn.% ↓Gen.% ↓
Base32.14100.00100.00100.00
Prompting25.00143.5964.79104.25
ASTC33.3381.5580.7094.83
ASTC (deletion)32.7488.8386.8599.58

We added the following experiments to demonstrate that Soft tokens aggregate information within each group under a limited length budget and preserve usable representations for subsequent reasoning.

① A control using Hard representatives of equal length tests whether aggregating information within a group is more effective than retaining a single original state under the same representation budget. With a shared input prefix, we keep the protection rules, grouping, number of representatives, and logical positions identical. The Hard version retains the hidden state of the last original token in each group, while ASTC aggregates the hidden states within the group.

② Masking subsequent reads further examines the role of representatives when they are read later. Starting from the next forward pass after the Soft representatives are generated, we block subsequent Attention from reading these representatives while retaining the physical KV, logical positions, and original GDN update rules.

MethodReadable by subsequent AttentionSuccessesAcc.% ↑D-KV% ↓Attn.% ↓Gen.% ↓
Base—54/16832.14100.00100.00100.00
Hard representatives of equal lengthYes52/16830.9589.2085.7299.08
ASTC SoftYes56/16833.3381.3780.5694.66
ASTC SoftNo46/16827.3889.1785.0699.25

Static long-context tests, three-attempt MBPP repair, … provide limited additional evidence for broad long-horizon robustness.

Thanks for your question. Table 1 of the original manuscript already demonstrates ASTC's strong performance on long-horizon agent tasks; this experiment aims to test whether ASTC's effectiveness extends to different task types, and the results show that its performance remains strong. In Appendix E of the original manuscript, we use RULER, LongBench v2, and interactive MBPP repair, following the evaluation approach of previous work on general-purpose context compression, to cover capabilities such as long-context retrieval, understanding, and code repair with feedback. Compared with previous work such as LCLM, ASTC reduces both Decode KV and Attention Pairs while maintaining task performance close to Base. Together with the performance on the agent benchmarks in the main text, these results support that ASTC's quality and efficiency gains extend across multiple task types, demonstrating strong robustness.

near-floor Kimi-Linear results provide limited additional evidence for broad long-horizon robustness.

Thank you for your question. This experiment is presented in Appendix G of the original manuscript, and we have also added results on Qwen3.8-27B; ASTC performs strongly on all three models. These results demonstrate that ASTC can improve quality across different backbones or provide broad long-horizon robustness guarantees.

ModelMethodAcc.% ↑D-KV% ↓Attn.% ↓Gen.% ↓
Qwen3.5-9BBase32.14100.00100.00100.00
Qwen3.5-9BASTC33.3381.5580.7094.83
Qwen3.8-27BBase78.57100.00100.00100.00
Qwen3.8-27BASTC77.9891.7064.43103.82
Kimi-Linear-48B-A3BBase1.47100.00100.00100.00
Kimi-Linear-48B-A3BASTC4.4184.0188.7694.70

Thank you again for your careful and specific feedback. To the best of our knowledge, ASTC is among the first training-free, in-model soft-token methods for context compression in long-horizon LLM agents. We hope that these point-by-point explanations and additional experiments address your concerns; if any points require further clarification, we would be happy to continue the discussion.

Reviewer_UsQcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ASTC, a training-free context compression method for LLM agents built on hybrid Gated Delta Network (GDN)–attention models. After a fixed early layer, it uses two signals from the GDN layers to select positions inside tool observations: an innovation ratio χ (how much of a token's value the preceding recurrent state does not predict) and an endpoint read error ψ. Selected positions are mean-pooled into a small number of soft tokens (at most 16 per tool observation in the AppWorld setting), protected exact values stay as hard tokens, and the later layers run on the shorter sequence with a reusable KV cache. The method is evaluated with Qwen3.5-9B on AppWorld, OfficeBench and 8-Objective QA, with additional results on RULER, LongBench v2, MBPP and Kimi-Linear.

Critical strengths
  1. Using signals the model already computes (the GDN delta update) to decide what to compress, without training a compressor, is a simple and interesting idea, and exact values are kept explicitly.
  2. The workload accounting includes auxiliary compressor calls, and the paper states the limits of its theory and metrics clearly.
Critical weaknesses
  1. The headline claims are stronger than the results. The abstract says ASTC "matches or exceeds the uncompressed agent's accuracy with up to 50.68% fewer end-to-end Attention Pairs". The 50.68% comes from OfficeBench (Attention Pairs 49.32% of Base, Table 1), where ASTC's accuracy is below Base (47.37 vs 48.42) and where only later byte-identical repetitions of an application-interface scaffold are compressed, while observations stay explicit (lines 980 to 983). The reduction is 19.30% on AppWorld and 1.49% on 8-Objective QA. The AppWorld accuracy gain (33.33 vs 32.14 on 168 tasks) is two tasks, and Table 1 gives a single value per method and metric, with no variance or confidence interval. The paper itself says the success advantage in the selector-control comparison is "suggestive rather than conclusive" (line 420), and both intervals in Figure 5 include zero. The "15.4×–103.7×" advantage over trained compressors is measured on AppWorld only and covers the three methods in Figure 2. Computed from Table 1 in the same way, ACON UT needs 15.1× and LLMLingua v1 12.3× the Attention Pairs per successful task, and both are marked as not training-free in Table 1. On 8-Objective QA, ACON UTCO is at or above Base (EM 28.54 vs 28.28, F1 37.40 vs 37.34) with 14.62% of Base Attention Pairs, while ASTC uses 98.51%.

  2. The efficiency results are analytical counts only. Decode KV and Attention Pairs "are analytical counts, not measured latency or GPU operations" (line 959), and no latency, throughput or memory measurement is reported. ASTC adds work that these counts do not cover: computing χ from the GDN state and ψ with a per-head reader, grouping and pooling, a post-cast error check, and a stable sort of the new suffix (Algorithm 1). The paper notes that "the avoided work must exceed selection and pooling cost for a net gain" (lines 343 to 344), but this cost is not measured. The caches of the layers before the boundary also keep their original length (lines 267 to 268). Measured latency, throughput and peak memory against Base are needed, especially on 8-Objective QA, where Attention Pairs fall by only 1.49%.

Questions for authors
  1. Can you report wall-clock latency, throughput and peak GPU memory for Base and ASTC on AppWorld, including the selection and pooling overhead?
  2. How were the protection rules, the thresholds and the compression stage chosen for each benchmark and model, and on which split? Which layer boundary does each stage in Figure 1 correspond to? The +1.19 pp point in Figure 1 equals the Table 1 test result (33.33 vs 32.14); was the stage chosen on the AppWorld test tasks?
  3. Which compressor models are used for LLMLingua v1 and LLMLingua-2, and how are their auxiliary calls converted into Attention Pairs?
  4. How many tasks do OfficeBench and 8-Objective QA contain, and can you give paired confidence intervals for Table 1, as in Figure 5?
  5. On 8-Objective QA, how does ASTC compare with ACON UTCO at a matched Attention Pairs budget?
4 comments
AuthorsOpenReview ↗

Response to Review UsQc(1/2)

Thanks very much for your thorough review and insightful questions. We will reply point-by-point below and hope to address your concerns!

Can you report wall-clock latency, throughput and peak GPU memory for Base and ASTC on AppWorld, including the selection and pooling overhead?

Thank you for your suggestion. We have added measurements on AppWorld and 8-Objective QA using an NVIDIA A100-SXM4-40GB. ASTC (Agentic Soft Token Compression) reduces both mean and P95 latency, shortens total queue time, and increases overall queue throughput.

DatasetMethodMean latency (s/task) ↓P95 (s/task) ↓token/s/GPU ↑Total queue time (min) ↓Overall queue token/s ↑Peak GPU memory (GiB) ↓Selection overheadPooling overheadTotal compression entry overhead
AppWorldBase73.69210.0824.8737.70136.1021.68———
AppWorldASTC71.00207.4724.4334.43141.0521.465.66413.29246.005
8-Objective QABase182.11339.2726.95——34.72———
8-Objective QAASTC157.62273.7725.54——36.97280.057144.705631.412

Total queue time on AppWorld decreases by 8.67%, and mean latency per task on 8-Objective QA decreases by 13.45%. We have added the relevant explanations and results to the appendix of the revised manuscript.

How were the protection rules, the thresholds and the compression stage chosen for each benchmark and model, and on which split? Which layer boundary does each stage in Figure 1 correspond to? The +1.19 pp point in Figure 1 equals the Table 1 test result (33.33 vs 32.14); was the stage chosen on the AppWorld test tasks?

Thank you for your question. ASTC's protection rules are part of the algorithm's design and are defined based on content types and field formats; for unprotected positions, the model's internal signals determine whether they are suitable for compression. The original manuscript used empirically set compression parameters and stages, and Stage 3 was determined empirically before the stage comparison in Figure 1, without using this test curve to search for the stage parameter. All layer indices are zero-based for Qwen3.5-9B.

Stage1234567
After GDN layer261014182226
Before Attention layer371115192327

This time, we conducted a small-scale search over χ and ψ on the AppWorld dev set and extended it to OfficeBench and 8-Objective QA, finding improvements in both quality and efficiency, demonstrating the algorithm's strong robustness and the potential to reach a higher performance ceiling through further parameter optimization in the future.

DatasetMethodAcc./EM% ↑F1% ↑D-KV% ↓Attn.% ↓Gen.% ↓
AppWorldBase32.14—100.00100.00100.00
AppWorldASTC (Tool)33.33—81.3780.5694.66
AppWorldASTC (Context)32.14—80.5281.8395.73
OfficeBenchBase48.42—100.00100.00100.00
OfficeBenchASTC (Tool)54.74—42.0940.1661.24
OfficeBenchASTC (Context)51.58—37.1140.7658.91
8-Objective QABase28.9137.51100.00100.00100.00
8-Objective QAASTC (Tool)30.8139.1857.9674.2982.02
8-Objective QAASTC (Context)31.3141.4066.8377.7587.40

Which compressor models are used for LLMLingua v1 and LLMLingua-2, and how are their auxiliary calls converted into Attention Pairs?

Thank you for raising this question. The comparisons of ASTC with baselines such as LLMLingua are fully fair: all configurations follow the detailed descriptions in the original LLMLingua papers, and the relevant details are documented in the appendix of the original manuscript; we have also made the code for reproducing the baselines and their training results available in our anonymous GitHub repository. LLMLingua v1 uses Qwen3.5-9B as the causal token scoring model; LLMLingua-2 uses the publicly available pretrained microsoft/llmlingua-2-xlm-roberta-large-meetingbank, a bidirectional token classifier with 24 layers. Both directly load pretrained models, with no additional parameter training in these baseline services. We will clarify the conversion of auxiliary calls to Attention Pairs using the following consistent accounting rules.

① LLMLingua v1: we accumulate valid query–key pairs across the 8 Full Attention layers. For a forward pass with q new positions and c cached positions, the count is 8 × [q·c + q(q+1)/2].

② LLMLingua-2: since the encoder uses bidirectional attention, the count is 24 × Σ_j n_j², calculated using the valid lengths of the actual windows.

AuthorsOpenReview ↗

Response to Review UsQc(2/2)

How many tasks do OfficeBench and 8-Objective QA contain, and can you give paired confidence intervals for Table 1, as in Figure 5?

Thank you for your suggestion. The configurations of all three datasets follow the official ACON implementation, and OfficeBench contains 46 parent task groups and 95 evaluation subtasks. 8-Objective QA uses a fixed set of 99 tasks with multiple objectives, each containing 8 question-answering objectives, for a total of 792 objectives. We have also added their confidence intervals, whose results show that ASTC has significant advantages in quality, computational cost, and generated token count; we have added the relevant explanations and results to the appendix of the revised manuscript.

DatasetComparisonΔAcc./EM (pp)ΔF1 (pp)D-KV%Attn.%Gen.%
OfficeBenchTool / Base+6.32 [−3.16, 16.84]—42.09 [24.29, 68.34]40.16 [26.39, 58.79]61.24 [45.92, 80.55]
OfficeBenchContext / Base+3.16 [−6.32, 12.63]—37.11 [21.75, 61.04]40.76 [26.59, 61.05]58.91 [43.66, 78.90]
8-Objective QATool / Base+1.89 [−2.78, 6.57]+1.67 [−4.07, 7.31]57.96 [32.74, 98.63]74.29 [54.52, 99.31]82.02 [64.41, 98.68]
8-Objective QAContext / Base+2.40 [−2.40, 7.32]+3.89 [−1.79, 9.51]66.83 [37.88, 118.23]77.75 [57.96, 104.03]87.40 [68.14, 109.92]

On 8-Objective QA, how does ASTC compare with ACON UTCO at a matched Attention Pairs budget?

Thank you for your suggestion. On 8-Objective QA, we added budget settings for ACON UTCO for comparison, reusing the existing optimized compression prompt; since the Attention cost cannot be strictly fixed, we selected comparisons where ASTC's Attention budget is similar to or even lower than ACON's. The results show that ASTC outperforms ACON UTCO.

MethodEM% ↑F1% ↑Total Attention (T) ↓Attn.% ↓D-KV% ↓Gen.% ↓
Base28.9137.512.0785100.00100.00100.00
ASTC30.8139.181.544274.2957.9682.02
ACON UTCO30.8140.211.745783.9993.20101.02

w1:The headline claims are stronger than the results...The 50.68% comes from OfficeBench (Attention Pairs 49.32% of Base, Table 1), where ASTC's accuracy is below Base and where only later byte-identical repetitions of an application-interface scaffold are compressed, while observations stay explicit. The reduction is 19.30% on AppWorld and 1.49% on 8-Objective QA. The AppWorld accuracy gain is two tasks, and Table 1 gives a single value per method and metric, with no variance or confidence interval...The "15.4×–103.7×" advantage over trained compressors is measured on AppWorld only and covers the three methods in Figure 2...On 8-Objective QA, ACON UTCO is at or above Base (EM 28.54 vs 28.28, F1 37.40 vs 37.34) with 14.62% of Base Attention Pairs, while ASTC uses 98.51%.

Thank you for your suggestion. As a training-free method, ASTC's empirical performance is consistent with the claims. In Table 1 of the revised manuscript, ASTC achieves quality improvements of varying magnitudes on all three datasets, while the baselines incur 12.3×–194.9× the Attention Pairs cost of ASTC on AppWorld and 18.0×–41.5× on OfficeBench; OfficeBench accuracy increases from 48.42% to 54.74%, with Attention Pairs reduced by 59.84%; on 8-Objective QA, EM/F1 increases from 28.91/37.51 to 30.81/39.18, with Attention Pairs reduced by 25.71%.

MethodAppWorldOfficeBench
Prompting1.1×2.0×
LLMLingua v112.3×18.0×
LLMLingua-2194.9×29.6×
ACON UT15.1×32.5×
ACON UTCO15.4×33.7×
LCLM103.9×41.5×

Thank you again for your careful and specific comments. To the best of our knowledge, ASTC is among the first training-free, in-model soft-token methods for context compression in long-horizon LLM agents. We hope that the revised manuscript and supplementary results address your concerns; if any points require further clarification, we would be happy to continue the discussion.

Dear Reviewer UsQc,

The authors have submitted their rebuttal and we are closing in on the end of the discussion period. Please take some time to carefully read over the rebuttal and respond whether your concerns have been addressed.

Best, AC

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer UsQc,

We would be very grateful to hear your thoughts on our responses and are eager to address any remaining concerns. Please let us know if there are any points you would like us to clarify or discuss further during the remaining discussion period. Thank you again for your time and detailed review!

Best regards,
Authors

Reviewer_GFqPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose to address the issue of long horizon LLMs through soft token compression. The compression recipe, called ASTC, is as follows: early layers process all tokens in the uncached suffix, then they have tests (innovation relative to the recurrent state; post-cast relative hidden-state error) at an intermediate point to determine which part of the context remains as hard tokens and which part becomes soft tokens, and finally later layers process a mix of the soft and hard tokens. Their analysis provides a bound on the compressor objective mismatch and offers conditions for ASTC to achieve lower distortion than hard-token representations of the same lengths, strengthening the paper. Their experiments cover Qwen3.5-9B evaluated on multiple benchamrks and report several useful metrics, in addition to the accuracy, such as the end-to-end decode KV, generated tokens and attention pairs. The authors show that ASTC preserves performance with up to 50% fewer attention pairs and have the advantage that their approach is training-free compared to other related works.

Critical strengths
  • I like the idea of using intrinsic quantities to the model to decide about what gets transformed into soft tokens and what remains as hard tokens.
  • The fact that the method does not require any training gives it an advantage compared to method that do requite training in that it's more compatible with continual learning and has less risk of overfitting to whatever data the compressor is usually trained on.
  • The theoretical analysis provides a strong intuition and understanding of the method.
  • The evaluation reports useful metrics beyond the accuracy and includes cache reuse experiments which are quite helpful to understand the advantages of the proposed approach.
Critical weaknesses
  • The paper is honestly hard to read when it comes to the method itself. The tests used can be simplified and explained much more clearly in the introduction. The algorithm is quite dense and barely explained or discussed in the main body of text. The mix between the method and the theoretical analysis makes the method section so hard to digest. The authors should definitely work on improving the clarity of their manuscript. Figure 1 should be in theory quite intuitive and easy to follow; and it should have a caption that makes it self-contained but it's neither intuitive in terms of understanding how the actual method works nor does it have a caption that helps understand it. In general the captions need substantial improvement. The authors need to fix the flow of the paper and its readability.
  • Several hard token baselines compress the full context whereas the proposed method has a hybrid context so for me this is not really a fair comparison, since the baselines would naturally lead to more degradation.
  • The details of the method are not explained sufficiently to be able to understand all the technicalities.
  • Component ablations show that hard deletion still keeps accuracy quite close to base, which makes me question how much this method adds. I would've expected a much sharper decline from this intervention so I'd want to see more convincing ablations.
Questions for authors
  • For LCLM, which compressor did you train? what are the technical details? is it matched to your method to be a fair comparison?
  • what's your compression target?
  • see last point above about the component ablations; do you have a more convincing demonstration of what the soft tokens add?
  • how was the boundary chosen in practice? what is fixed or did you use some sort of train/validation split to detect it?
4 comments
AuthorsOpenReview ↗

Response to Review GFqP(1/2)

Thanks very much for your thorough review and insightful questions. We will reply point-by-point below and hope to address your concerns!

For LCLM, which compressor did you train? what are the technical details? is it matched to your method to be a fair comparison?

Thank you for raising this question. The comparisons of ASTC (Agentic Soft Token Compression) with baselines such as LCLM, ACON, and LLMLingua are fully fair: all methods were configured according to the detailed descriptions in their respective original papers, and the relevant details are documented in the appendix of the original manuscript; we have also made the code for reproducing the baselines and their training results available in our anonymous GitHub repository. For LCLM, we use the publicly available pretrained latent-context/0.6b-4b-LCLM-4x compressor, freeze its 0.6B encoder and existing 1024→2560 adapter, and train only a newly added 2560→4096 MLP Bridge and LoRA on the Qwen decoder to adapt the compressed representations to the same initial Qwen3.5-9B backbone used by ASTC. The original Qwen backbone parameters remain frozen, and LoRA uses rank=8, alpha=16, and dropout=0.05. Adaptation training uses interaction trajectories to construct next-action prediction samples, first using action-prediction and auxiliary recap supervision, followed by action prediction aligned with the history input format used in practice; the optimizer is AdamW, with a learning rate of 2×10^-4. During inference, older history is compressed at approximately 4×, while recent interactions and interactions added after compression are retained verbatim. We believe that the performance difference between LCLM and ASTC mainly stems from their different research focuses: LCLM primarily studies general-purpose context compression and lists compression of an agent's accumulated working history as future work, whereas ASTC directly addresses history compression during ongoing interactions through training-free selection and aggregation within the model.

what's your compression target?

Do you mean what is compressed or the degree of compression? We compress the historical context accumulated by an agent during ongoing interactions and read for subsequent decisions; each step processes only newly added segments, and the resulting compressed KV entries are continuously reused in subsequent generation and interaction steps. At the same time, we do not prescribe a single compression ratio for all tasks, and the actual merging scope and retained length vary dynamically with the input content. This paper evaluates compression using both task quality and end-to-end workload.

see last point above about the component ablations; do you have a more convincing demonstration of what the soft tokens add?

Thank you for raising this question. Hard deletion in the ablation study is actually a variant of ASTC that uses ASTC's protection and selection rules but directly deletes redundant, uninformative positions without generating Soft tokens. Therefore, it still benefits from ASTC's internal selection and protection. The direct-deletion method you refer to is the Prompting baseline in Table 1. We have clearly distinguished the two in the revised manuscript.

VariantAcc.% ↑D-KV% ↓Attn.% ↓Gen.% ↓
Base32.14100.00100.00100.00
Prompting25.00143.5964.79104.25
ASTC33.3381.5580.7094.83
ASTC (deletion)32.7488.8386.8599.58

We added the following experiments to demonstrate that Soft tokens aggregate information within each group under a limited length budget and preserve usable representations for subsequent reasoning.

① A control using Hard representatives of equal length tests whether aggregating information within a group is more effective than retaining a single original state under the same representation budget. With a shared input prefix, we keep the protection rules, grouping, number of representatives, and logical positions identical. The Hard version retains the hidden state of the last original token in each group, while ASTC aggregates the hidden states within the group.

② Masking subsequent reads further examines the role of representatives when they are read later. Starting from the next forward pass after the Soft representatives are generated, we block subsequent Attention from reading these representatives while retaining the physical KV, logical positions, and original GDN update rules.

MethodReadable by subsequent AttentionSuccessesAcc.% ↑D-KV% ↓Attn.% ↓Gen.% ↓
Base—54/16832.14100.00100.00100.00
Hard representatives of equal lengthYes52/16830.9589.2085.7299.08
ASTC SoftYes56/16833.3381.3780.5694.66
ASTC SoftNo46/16827.3889.1785.0699.25
AuthorsOpenReview ↗

Response to Review GFqP(2/2)

how was the boundary chosen in practice? what is fixed or did you use some sort of train/validation split to detect it?

Thank you for your question. The original manuscript used empirically set compression parameters. This time, we conducted a small-scale search over χ and ψ on the dev set and found improvements in both quality and efficiency on OfficeBench and 8-Objective QA. Meanwhile, the parameter choices across the three benchmarks are similar, with some even being identical, demonstrating the algorithm's strong robustness and the potential to reach a higher performance ceiling through further parameter optimization in the future.

DatasetMethodAcc./EM% ↑F1% ↑D-KV% ↓Attn.% ↓Gen.% ↓
OfficeBenchOriginal manuscript47.37—54.5249.3269.23
OfficeBenchRevised54.74—42.0940.1661.24
8-Objective QAOriginal manuscript30.5638.4678.6998.5186.55
8-Objective QARevised30.8139.1857.9674.2982.02

The paper is honestly hard to read when it comes to the method itself. ... The authors need to fix the flow of the paper and its readability.

Thanks for your careful review. In the revised manuscript, we explain the logic and rules of the entire algorithm in intuitive language in the introduction. We also redrew Figure 1 to clearly show the relationships among the algorithm's components. We further expanded the caption to clarify the notation, the retention branch, and the relationship between the left and right subfigures. We also organized the method and theory separately and carefully explained the algorithm's ideas and procedure in the main text to help readers understand. Building on the original explanations, the revised manuscript adds the experimental details and explanations you requested.

Several hard token baselines compress the full context whereas the proposed method has a hybrid context so for me this is not really a fair comparison, since the baselines would naturally lead to more degradation.

Thank you for pointing out this factor. The baselines mainly compress the full context, but we observed that a large portion of an agent's context is generated by tools, and this context both contains compressible redundant information and is challenging to compress; we therefore first applied compression to tool outputs. For a fair comparison, we compared full-context compression in Appendix F of the original manuscript and found comparable results.

DatasetMethodAcc./EM% ↑F1% ↑D-KV% ↓Attn.% ↓Gen.% ↓
AppWorldBase32.14—100.00100.00100.00
AppWorldASTC (Tool)33.33—81.3780.5694.66
AppWorldASTC (Context)32.14—80.5281.8395.73
OfficeBenchBase48.42—100.00100.00100.00
OfficeBenchASTC (Tool)54.74—42.0940.1661.24
OfficeBenchASTC (Context)51.58—37.1140.7658.91
8-Objective QABase28.9137.51100.00100.00100.00
8-Objective QAASTC (Tool)30.8139.1857.9674.2982.02
8-Objective QAASTC (Context)31.3141.4066.8377.7587.40

Thank you again for your careful and specific feedback. To the best of our knowledge, ASTC is among the first training-free, in-model soft-token methods for context compression in long-horizon LLM agents. We hope that the revised manuscript and supplementary results address your concerns; if any points require further clarification, we would be happy to continue the discussion.

Thanks for the clarifications and additional experiments, I think the matched-length experiments help, although the gain over ASTC deletion remains small, so the additional value of soft tokens is still not fully clear. The LCLM comparison also remains difficult to interpret because of the different action limits and training/inference input formats. Overall, I find the training-free approach interesting and the results promising, so I keep my current positive score.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for following up and for your constructive feedback. We are pleased that our previous responses were helpful.

I think the matched-length experiments help, although the gain over ASTC deletion remains small, so the additional value of soft tokens is still not fully clear.

The read-masking experiment supported the usefulness of the information retained in soft tokens for subsequent decisions. This experiment preserved their physical KV entries, logical positions, and original recurrent-state update rules, while blocking subsequent attention from reading them starting with the next forward pass. The success rate fell from 33.33% to 27.38%. This result supported the role of soft tokens as compact, attention-accessible representations that aggregate redundant content for use in subsequent interactions.

The benefits of soft tokens were also reflected in cumulative workload over the full task trajectory. In these autonomous runs, this workload depended on subsequent generation length and interaction trajectories. In Table 2, compared with ASTC (deletion), full ASTC required 7.08% fewer total Attention Pairs, 8.20% fewer Decode KV interactions, and 4.77% fewer generated tokens. We hypothesize that deleting compressible content may also discard cues that remain useful later, increasing the work required for subsequent reasoning or interaction. By aggregating and retaining these cues, soft tokens may help reduce this additional workload, consistent with the observed reductions in generated tokens and cumulative attention workload.

The LCLM comparison also remains difficult to interpret because of the different action limits and training/inference input formats.

Thank you for this valuable feedback. We previously matched the action limits and aligned the training/inference input formats, but LCLM performed poorly under that configuration. To improve the baseline’s performance on the target tasks and avoid underestimating its capabilities, we adjusted LCLM’s configuration and used the better-performing settings. Note that LCLM mainly evaluates on reasoning tasks instead of agentic tasks, which focuses on merely chain-of-thought compaction. The concurrent work, LCLM paper, also highlights a future direction to “extend online latent compaction beyond chain-of-thought to tool observations and an agent’s accumulated working history.” This difference in research scope between LCLM and ASTC help explain LCLM’s weaker performance on our agent tasks. The overall results showed that ASTC still had a substantial advantage on these tasks.

DatasetMethodAcc./EM% ↑F1% ↑D-KV% ↓Attn.% ↓Gen.% ↓
AppWorldBase32.14—100.00100.00100.00
AppWorldLCLM (Adapted)6.55—23.941643.6535.08
AppWorldLCLM (Fully Fair)0.00—8.66167.4641.04
AppWorldASTC (Tools)33.33—81.3780.5694.66
AppWorldASTC (Context)32.14—80.5281.8395.73
OfficeBenchBase48.42—100.00100.00100.00
OfficeBenchLCLM (Adapted)21.05—20.83641.3033.37
OfficeBenchLCLM (Fully Fair)11.58—17.5189.6757.15
OfficeBenchASTC (Tools)54.74—42.0940.1661.24
OfficeBenchASTC (Context)51.58—37.1140.7658.91
8-Objective QABase28.9137.51100.00100.00100.00
8-Objective QALCLM (Adapted)0.000.0022.470.830.20
8-Objective QALCLM (Fully Fair)0.130.137.6631.8418.48
8-Objective QAASTC (Tools)30.8139.1857.9674.2982.02
8-Objective QAASTC (Context)31.3141.4066.8377.7587.40

Thank you for your guidance and for the time and effort you have invested throughout this process. We would be happy to discuss any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Contributions

Dear reviewers and chairs,

We sincerely thank all reviewers for their positive and insightful feedback. We are encouraged by their recognition of our idea as “interesting” (Reviewers GFqP and UsQc) and their shared appreciation of our training-free design. Our method uses “signals the model already computes” to decide what to compress (Reviewer UsQc). Compared with methods requiring training, this design is “more compatible with continual learning” and has “less risk of overfitting to whatever data the compressor is usually trained on” (Reviewer GFqP). We also appreciate the acknowledgement of our “promising” results (Reviewer GFqP) and “useful experimental controls” (Reviewer KkTX), and the recognition of our approach as “a practical direction” for reducing agent-context computation (Reviewer KkTX).

We addressed the reviewers’ questions point by point and incorporated the clarifications and revisions into the final version of the paper. In response to Reviewer GFqP, we provided a deeper discussion of the role of agentic soft tokens and improved the manuscript’s readability. To address the main concerns raised by Reviewers UsQc and KkTX regarding comparison fairness, generalization, and practical efficiency, we further refined our baseline comparisons, extended our evaluations of generalization and robustness, and provided a more detailed analysis and discussion of practical efficiency in the revised manuscript.

Our work introduces a training-free, in-model soft-token compression method for long-horizon agent tasks, and we believe it offers a valuable direction for context compression research. We sincerely appreciate the time and effort invested by all reviewers and chairs.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.