- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed CCM. At every turn the agent rewrites a compact memory, and the next prompt holds only the task, that memory and the newest observation. It first runs CCM without training on four frontier models on TerminalBench-2. It then trains on WebShop and Endless Terminals with GRPO plus an auxiliary action-token distillation loss. In that loss, a frozen copy of the initial model sees the full history and acts as the privileged teacher. The paper reports that the auxiliary loss clearly raises CCM success over GRPO alone.
Critical strengths
S1, The paper addresses a pivotal and highly timely challenge in long-horizon LLM agent deployment, i.e., mitigating KV cache growth and prompt overhead without sacrificing task performance.
S2, The manuscript is commendably transparent and reproducible, offering exhaustive disclosures of prompt templates, hyperparameter settings, and turn-by-turn token accounting.
S3, The proposed combination of CCM, GRPO, and auxiliary history distillation yields an impressive empirical gain on Qwen3-4B in WebShop, widening the success rate from 32.8% to 86.7% over the baseline.
Critical weaknesses
W1, The core motivation of saving memory and compute is unmeasured, as the paper provides zero metrics for GPU KV cache, latency, or throughput.
W2, In practice, CCM increases turn counts by 20–45% and output tokens by 40–180%. Furthermore, by invalidating prefix caching, it theoretically increases inference latency.
W3, Non-trained inference performance drops on 3 out of 4 models, and Kimi’s apparent gain of 1 task is statistically insignificant.
W4, Unfair comparison: The untrained full-history baseline is capped at 4,096 tokens, while the teacher is given a 32,768 token limit.
W5, Standard compaction baselines (such as Terminus-2 default threshold summarization) and simple observation masking are omitted from experimental comparisons.
W6, The problem formulation is essentially identical to MEM1, and no direct empirical comparison against MEM1 is provided despite overlapping benchmarks.
W7, The mechanism behind the performance gain is unverified, as an untrained base teacher with only 4–16% accuracy is unlikely to provide rich guidance; the improvement is more plausibly driven by policy anchoring that prevents GRPO collapse.
W8, Critical ablations are missing, including testing a teacher restricted to compact context, tuning reference KL penalties, and testing a memoryless baseline.
W9, Training efficiency is ignored, masking the heavy computational cost of requiring a full-history forward pass through the teacher model at every step.
Questions for authors
Q1: Can you provide quantitative metrics for GPU KV cache memory, wall-clock latency, and throughput to support the claim that CCM is memory- and compute-efficient?
Q2: Why was the untrained full-history baseline capped at 4,096 tokens while the teacher received 32,768 tokens, and can you report results for an untruncated full-history baseline?
Q3: Why was checkpoint selection performed directly on the test set, and can you provide results evaluated over multiple seeds with confidence intervals?
Q4: How does your method differ empirically and conceptually from MEM1, and why is a direct experimental comparison with MEM1 omitted?
Q5: Can you run a memoryless baseline (task + current observation only) trained with GRPO to prove that compact memory is necessary for WebShop?
Q6: What is the total training compute and wall-clock time required for the full-history teacher forward passes during GRPO training?
4 comments
Q4: How does your method differ empirically and conceptually from MEM1, and why is a direct experimental comparison with MEM1 omitted?
We have now added a MEM1-style PPO baseline for both WebShop model sizes. MEM1 and CCM share the principle of maintaining a compact state across interactions rather than retaining the complete history. Their training procedures differ: MEM1 uses PPO with a learned critic, whereas our method combines critic-free GRPO with an auxiliary objective that supervises compact-context actions using a teacher conditioned on full interaction history.
MEM1 implements its changing-context policy and value computations using custom full-trajectory attention masks. Ours instead reconstructs each turn's actual retained context for standard trainer engines. This preserves the intended per-turn conditioning without requiring MEM1's custom attention implementation.
We select each checkpoint using the separate 100-task validation set, breaking ties in favor of the earlier checkpoint. The table below reports success and context metrics from that checkpoint's fixed 128-task test evaluation, with seed 42, temperature 0.4, a 4096-token input cap and 1024 output tokens per turn.
| Model | Method | Selected step | Test success | Memory update | Prefix retention | Mean memory tokens | Max memory tokens | Median peak prompt tokens | Max peak prompt tokens |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3-4B-Instruct | CCM + GRPO + distillation | 100 | 86.72% | 100.0% | 77.2% | 60.1 | 89 | 1041.5 | 1279 |
| Qwen3-4B-Instruct | MEM1-style PPO | 100 | 78.13% | 99.2% | 2.2% | 33.5 | 109 | 1380 | 1643 |
| Qwen3-8B | CCM + GRPO + distillation | 60 | 75.00% | 92.3% | 83.1% | 194.5 | 337 | 1141.5 | 1471 |
| Qwen3-8B | MEM1-style PPO | 100 | 80.47% | 99.3% | 2.3% | 117.3 | 448 | 1429.5 | 1758 |
The success comparison is mixed: CCM + distillation performs better on 4B, whereas MEM1-style PPO performs better on 8B. MEM1-style PPO has shorter mean carried summaries, while CCM + distillation has smaller median and maximum peak prompts in both model sizes.
Both methods update memory on most turns, but CCM + distillation retains a substantially larger token prefix between consecutive memories.
Our method does not require a learned critic or critic optimization. Instead, it adds full-history teacher scoring through an additional forward pass.
Q5 Memoryless GRPO Baseline
Q5: Can you run a memoryless baseline (task + current observation only) trained with GRPO to prove that compact memory is necessary for WebShop?
We trained memoryless GRPO baselines on WebShop for Qwen3-4B-Instruct and Qwen3-8B, including thinking-enabled and thinking-disabled variants for 8B. Each turn receives the original task, current observation and available actions, but no earlier actions, earlier observations or persistent generated memory. In the thinking-enabled variant, the model may reason before its action, but that reasoning is not carried into the next turn.
We select checkpoints using success on the separate 100-task validation set, breaking ties in favor of the earlier checkpoint, and report success on the fixed 128-task test set. Evaluation uses seed 42, temperature 0.4, a 4096-token input cap, a 1024-token total output budget per turn and at most 15 actions.
| Model | Method | Validation-selected step | Validation success | Test success |
|---|---|---|---|---|
| Qwen3-4B-Instruct | Memoryless GRPO | 90 | 3% | 2.34% |
| Qwen3-4B-Instruct | Full-history GRPO | 100 | 79% | 75.78% |
| Qwen3-4B-Instruct | CCM + GRPO + distillation | 100 | 83% | 86.72% |
| Qwen3-8B | Memoryless GRPO, thinking enabled | 80 | 3% | 2.34% |
| Qwen3-8B | Memoryless GRPO, thinking disabled | 50 | 2% | 3.13% |
| Qwen3-8B | Full-history GRPO | 80 | 84% | 82.81% |
| Qwen3-8B | CCM + GRPO + distillation | 60 | 76% | 75.00% |
Q6: What is the total training compute and wall-clock time required for the full-history teacher forward passes during GRPO training?
Using Webshop, we report the full-history teacher-scoring cost over completed CCM + GRPO + Distill runs:
| Model | FLOPs | Teacher scoring time | Total training time | Teacher time share |
|---|---|---|---|---|
| Qwen3-4B-Instruct | 3.13 x 10^18 | 1h28m00s | 10h47m51s | 13.6% |
| Qwen3-8B | 5.96 x 10^18 | 2h18m47s | 24h34m54s | 9.41% |
Thank you for the additional experiments and clarifications. They address some of my concerns, but my main concern about the practical benefit of CCM remains unresolved.
The distinction I care about is between reducing retained context and improving end-to-end efficiency. Your response explicitly states that CCM has higher task-attempt completion latency and higher net training time in the measured WebShop 8B workload. Lower live KV occupancy and lower TTFT therefore do not establish faster task completion or more efficient RL training. Similarly, higher output-token throughput is not sufficient evidence of useful throughput when the method itself generates substantially more text.
The cache accounting in Appendix A makes this tradeoff particularly relevant. The reported cache-read share decreases from 90.7% to 35.6% for Sonnet, from 88.3% to 0.0% for Opus, and from 92.6% to 42.3% for Kimi. Thus, the reduction in context length comes with substantially reduced cache reuse in these configurations. What remains unclear is whether the memory savings compensate for this loss of reuse and the additional generation overhead under a practical deployment or training budget.
One plausible application is a memory-constrained setting in which smaller active contexts permit more concurrent trajectories. However, increased concurrency does not necessarily translate into higher end-to-end throughput. In my view, retaining full history while making effective use of prefix caching, tuning concurrency to the available memory, or using lower-precision KV caching such as FP8 where supported are more conventional ways to address this constraint. I suspect these approaches may offer a better practical tradeoff than repeatedly rewriting memory, especially given the reduced cache reuse and additional generation reported here. The current evidence has not convinced me that CCM offers a practical advantage over these alternatives.
For inference, useful measurements would include completed and successfully solved tasks per unit time, together with success rate. For RL training, I would like to see wall-clock time or GPU-hours required to reach a comparable held-out success rate, including the full-history teacher-scoring overhead. Lower memory occupancy alone does not establish a training-efficiency advantage.
I also acknowledge that adding distillation improves CCM over CCM trained with GRPO alone, particularly on WebShop. However, this does not establish that CCM is preferable to a trained full-history agent. In the supplied manuscript, CCM with distillation exceeds full-history GRPO on WebShop 4B, but falls below it on WebShop 8B and Endless Terminals. Your new MEM1-style comparison is likewise mixed across model sizes.
My remaining concern is therefore not whether CCM can reduce retained context, which the experiments demonstrate, or whether distillation can improve CCM, which the WebShop results support. It is whether the proposed approach offers a demonstrated practical advantage over competitive alternatives. Please identify the operating regime in which it does, and evaluate that regime directly. The current evidence does not yet establish an end-to-end throughput or RL training-efficiency benefit.
Overall, I recognize the contribution of the proposed training loss, particularly its improvement over CCM trained with GRPO alone. However, these gains do not translate into a consistent improvement over full-history GRPO. I remain unconvinced about the practical application of the overall approach. The reported increase in net training time, together with reduced cache reuse and additional generation overhead, suggests that CCM may worsen rather than improve end-to-end RL training throughput in the evaluated setting. Although this does not establish that CCM is unhelpful in every setting, the paper has not demonstrated a practical benefit that offsets these costs. I therefore do not find the demonstrated contribution sufficient to justify acceptance.
Thank you for clarifying the distinction between reduced retained context and end-to-end efficiency. We agree that lower KV occupancy, lower time to first token, or higher output-token throughput alone do not establish faster task completion or more efficient RL training. Our measured 8B training workload does not demonstrate a training-efficiency advantage.
Direct Inference Measurements
We have now completed inference profiling of the validation-selected WebShop 4B checkpoint, measuring completed attempts and successfully solved tasks per unit time rather than token throughput alone.
| Metric | Full-history GRPO | CCM + GRPO + Distill |
|---|---|---|
| Test success | 78.13% (100/128) | 82.81% (106/128) |
| Collection wall time | 449.43 s | 287.24 s |
| Mean task-attempt latency | 9.46 s | 7.68 s |
| Completed attempts/hour | 1,025.3 | 1,604.2 |
| Successfully solved tasks/hour | 801.0 | 1,328.5 |
| Mean time to first token | 57.02 ms | 43.07 ms |
| Mean live KV/GPU | 0.3405 GiB | 0.2716 GiB |
| Observed peak live KV/GPU | 1.3601 GiB | 0.6350 GiB |
| Total generated tokens, including memory and actions | 39,883 | 56,783 |
| Inference TFLOPs/task | 72.98 | 30.74 |
In this configuration, CCM generates more output tokens but nevertheless completes the task set faster, with 65.9% higher successful-task throughput, 36.1% lower collection time, and 53.3% lower observed peak live KV occupancy.
Additional training cost could potentially be amortized through repeated inference if serving costs are lower as shown here.
Context Management And Systems Optimizations
We agree that cache-friendly prompt layouts, concurrency tuning, and lower-precision KV caching are important optimizations. Our work addresses a complementary question: can the agent learn what information to retain and how to update it as new observations arrive? Systems optimizations improve how a given context is stored and processed; model-controlled context management additionally changes what that context contains. By maintaining an explicit, inspectable memory, this approach leverages the agent’s own ability to organize task-relevant information, with the goal of improving long-horizon efficiency and performance.
Model-controlled context management is an active research direction, with related work reporting benefits on long-horizon agent tasks. Examples include optimized observation/history compression in ACON, joint training of task execution and summary generation in CompactionRL, and autonomous context editing and retrieval in ACM. These findings support its potential practical value.
Cache Reuse
We acknowledge that rewriting context can reduce cache reuse and increase generation costs. Opus's particularly large cache-read drop may also reflect Bedrock's higher minimum cacheable prefix: 4,096 tokens for Opus 4.6 versus 1,024 for Sonnet 4.6. This may help explain Opus's zero cache-read share, but does not remove the observed cache-reuse tradeoff.
Thank you for the thoughtful feedback. We respond to your concerns below.
Webshop 8B run
CCM produces substantially more text, so higher token throughput does not imply faster task completion. In fact, CCM's task-attempt finishing latency and net training time are higher here.
These results support lower observed live KV occupancy and lower TTFT in the measured CCM workload.
Thank you for highlighting this concern. We agree that an untruncated baseline is an important control.
We therefore evaluated the same untrained full-history models with the input limit increased to 32,768 tokens, retaining the fixed 128-task test set, 1024-token output cap, 15-action limit, temperature 0.4 and seed 42.
The increased limit eliminated truncation in both untrained controls but did not improve their observed success.
We now report validation-selected results instead: for each method, we select the checkpoint with the highest success on the separate 100-task validation set, breaking ties in favor of the earlier checkpoint, and report test success from that same checkpoint. The test sets contain 128 tasks for WebShop and 300 tasks for Endless Terminals.
Due to compute constraints, we could not run multiple independent training seeds.