acceptodds
Under review as a conference paper at ICLR 2027

Continuous Context Management

Abstract

Long-horizon large language model (LLM) agents commonly retain their complete interaction history until compaction is triggered at a predefined threshold. We study Continuous Context Management (CCM), which performs compaction at every turn to prevent interaction history from accumulating in the active prompt. At each turn, a CCM agent emits an updated memory together with an environment action; its next prompt contains the original task, retained memory, and newest observation rather than the complete transcript. We first evaluate CCM without fine-tuning on TerminalBench-2 using Claude Sonnet 4.6, Claude Opus 4.6, GLM-5, and Kimi K3. CCM substantially reduces cumulative input usage and active-prompt size, although it lowers task success for most models while preserving performance for Kimi K3. We use GRPO with privileged full-history distillation to improve CCM in open-weight models. A frozen copy of the student's initial model scores each sampled student action under the complete history reconstructed from that student's rollout, providing dense action-token supervision without a separate teacher rollout or reference solution. On WebShop, this objective substantially improves CCM over GRPO at both evaluated model scales and surpasses full-history GRPO for Qwen3-4B-Instruct, though not for Qwen3-8B. On Endless Terminals, the augmented method provides a modest improvement over GRPO, with both CCM policies outperforming the untrained full-history baseline. These results demonstrate that CCM is a viable inference paradigm for agents operating with substantially reduced retained context and that its performance can be improved through reinforcement learning with privileged full-history distillation.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Compaction, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_g133OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed CCM. At every turn the agent rewrites a compact memory, and the next prompt holds only the task, that memory and the newest observation. It first runs CCM without training on four frontier models on TerminalBench-2. It then trains on WebShop and Endless Terminals with GRPO plus an auxiliary action-token distillation loss. In that loss, a frozen copy of the initial model sees the full history and acts as the privileged teacher. The paper reports that the auxiliary loss clearly raises CCM success over GRPO alone.

Critical strengths

S1, The paper addresses a pivotal and highly timely challenge in long-horizon LLM agent deployment, i.e., mitigating KV cache growth and prompt overhead without sacrificing task performance.

S2, The manuscript is commendably transparent and reproducible, offering exhaustive disclosures of prompt templates, hyperparameter settings, and turn-by-turn token accounting.

S3, The proposed combination of CCM, GRPO, and auxiliary history distillation yields an impressive empirical gain on Qwen3-4B in WebShop, widening the success rate from 32.8% to 86.7% over the baseline.

Critical weaknesses

W1, The core motivation of saving memory and compute is unmeasured, as the paper provides zero metrics for GPU KV cache, latency, or throughput.

W2, In practice, CCM increases turn counts by 20–45% and output tokens by 40–180%. Furthermore, by invalidating prefix caching, it theoretically increases inference latency.

W3, Non-trained inference performance drops on 3 out of 4 models, and Kimi’s apparent gain of 1 task is statistically insignificant.

W4, Unfair comparison: The untrained full-history baseline is capped at 4,096 tokens, while the teacher is given a 32,768 token limit.

W5, Standard compaction baselines (such as Terminus-2 default threshold summarization) and simple observation masking are omitted from experimental comparisons.

W6, The problem formulation is essentially identical to MEM1, and no direct empirical comparison against MEM1 is provided despite overlapping benchmarks.

W7, The mechanism behind the performance gain is unverified, as an untrained base teacher with only 4–16% accuracy is unlikely to provide rich guidance; the improvement is more plausibly driven by policy anchoring that prevents GRPO collapse.

W8, Critical ablations are missing, including testing a teacher restricted to compact context, tuning reference KL penalties, and testing a memoryless baseline.

W9, Training efficiency is ignored, masking the heavy computational cost of requiring a full-history forward pass through the teacher model at every step.

Questions for authors

Q1: Can you provide quantitative metrics for GPU KV cache memory, wall-clock latency, and throughput to support the claim that CCM is memory- and compute-efficient?

Q2: Why was the untrained full-history baseline capped at 4,096 tokens while the teacher received 32,768 tokens, and can you report results for an untruncated full-history baseline?

Q3: Why was checkpoint selection performed directly on the test set, and can you provide results evaluated over multiple seeds with confidence intervals?

Q4: How does your method differ empirically and conceptually from MEM1, and why is a direct experimental comparison with MEM1 omitted?

Q5: Can you run a memoryless baseline (task + current observation only) trained with GRPO to prove that compact memory is necessary for WebShop?

Q6: What is the total training compute and wall-clock time required for the full-history teacher forward passes during GRPO training?

4 comments

Thank you for the thoughtful feedback. We respond to your concerns below.

Q1: Can you provide quantitative metrics for GPU KV cache memory, wall-clock latency, and throughput to support the claim that CCM is memory- and compute-efficient?

Webshop 8B run

MetricFull-history GRPOCCM + GRPO + Distill
Observed peak live KV-cache per GPU2.46 GiB2.21 GiB
Mean time to first token208 ms154 ms
Mean inter-token latency4.42 ms4.51 ms
Mean task-attempt finishing latency, including unsuccessful attempts7.60 s28.45 s
Aggregate rollout output throughput838 tokens/s2542 tokens/s
Net training wall-clock time13h47m13s24h34m54s

CCM produces substantially more text, so higher token throughput does not imply faster task completion. In fact, CCM's task-attempt finishing latency and net training time are higher here.

These results support lower observed live KV occupancy and lower TTFT in the measured CCM workload.

Q2: Why was the untrained full-history baseline capped at 4,096 tokens while the teacher received 32,768 tokens, and can you report results for an untruncated full-history baseline?

Thank you for highlighting this concern. We agree that an untruncated baseline is an important control.

We therefore evaluated the same untrained full-history models with the input limit increased to 32,768 tokens, retaining the fixed 128-task test set, 1024-token output cap, 15-action limit, temperature 0.4 and seed 42.

Untrained full-history modelTest success, 4096-token input capTest success, 32,768-token input capEpisodes truncated at 32KMaximum prompt at 32K
Qwen3-4B-Instruct3.91%3.91%0/1286873 tokens
Qwen3-8B16.41%15.63%0/1289488 tokens

The increased limit eliminated truncation in both untrained controls but did not improve their observed success.

Q3: Why was checkpoint selection performed directly on the test set, and can you provide results evaluated over multiple seeds with confidence intervals?

We now report validation-selected results instead: for each method, we select the checkpoint with the highest success on the separate 100-task validation set, breaking ties in favor of the earlier checkpoint, and report test success from that same checkpoint. The test sets contain 128 tasks for WebShop and 300 tasks for Endless Terminals.

Benchmark and modelMethodSelected stepValidation successTest success
WebShop, Qwen3-4B-InstructFull-history GRPO10079.0%75.78%
WebShop, Qwen3-4B-InstructCCM-GRPO9032.0%21.09%
WebShop, Qwen3-4B-InstructCCM-GRPO + Distill10083.0%86.72%
WebShop, Qwen3-8BFull-history GRPO8084.0%82.81%
WebShop, Qwen3-8BCCM-GRPO5032.0%25.00%
WebShop, Qwen3-8BCCM-GRPO + Distill6076.0%75.00%
Endless Terminals, Qwen3-8BFull-history GRPO3041.0%35.33%
Endless Terminals, Qwen3-8BCCM-GRPO3031.0%29.00%
Endless Terminals, Qwen3-8BCCM-GRPO + Distill5037.0%32.00%

Due to compute constraints, we could not run multiple independent training seeds.

Q4: How does your method differ empirically and conceptually from MEM1, and why is a direct experimental comparison with MEM1 omitted?

We have now added a MEM1-style PPO baseline for both WebShop model sizes. MEM1 and CCM share the principle of maintaining a compact state across interactions rather than retaining the complete history. Their training procedures differ: MEM1 uses PPO with a learned critic, whereas our method combines critic-free GRPO with an auxiliary objective that supervises compact-context actions using a teacher conditioned on full interaction history.

MEM1 implements its changing-context policy and value computations using custom full-trajectory attention masks. Ours instead reconstructs each turn's actual retained context for standard trainer engines. This preserves the intended per-turn conditioning without requiring MEM1's custom attention implementation.

We select each checkpoint using the separate 100-task validation set, breaking ties in favor of the earlier checkpoint. The table below reports success and context metrics from that checkpoint's fixed 128-task test evaluation, with seed 42, temperature 0.4, a 4096-token input cap and 1024 output tokens per turn.

ModelMethodSelected stepTest successMemory updatePrefix retentionMean memory tokensMax memory tokensMedian peak prompt tokensMax peak prompt tokens
Qwen3-4B-InstructCCM + GRPO + distillation10086.72%100.0%77.2%60.1891041.51279
Qwen3-4B-InstructMEM1-style PPO10078.13%99.2%2.2%33.510913801643
Qwen3-8BCCM + GRPO + distillation6075.00%92.3%83.1%194.53371141.51471
Qwen3-8BMEM1-style PPO10080.47%99.3%2.3%117.34481429.51758

The success comparison is mixed: CCM + distillation performs better on 4B, whereas MEM1-style PPO performs better on 8B. MEM1-style PPO has shorter mean carried summaries, while CCM + distillation has smaller median and maximum peak prompts in both model sizes.

Both methods update memory on most turns, but CCM + distillation retains a substantially larger token prefix between consecutive memories.

Our method does not require a learned critic or critic optimization. Instead, it adds full-history teacher scoring through an additional forward pass.

Q5 Memoryless GRPO Baseline

Q5: Can you run a memoryless baseline (task + current observation only) trained with GRPO to prove that compact memory is necessary for WebShop?

We trained memoryless GRPO baselines on WebShop for Qwen3-4B-Instruct and Qwen3-8B, including thinking-enabled and thinking-disabled variants for 8B. Each turn receives the original task, current observation and available actions, but no earlier actions, earlier observations or persistent generated memory. In the thinking-enabled variant, the model may reason before its action, but that reasoning is not carried into the next turn.

We select checkpoints using success on the separate 100-task validation set, breaking ties in favor of the earlier checkpoint, and report success on the fixed 128-task test set. Evaluation uses seed 42, temperature 0.4, a 4096-token input cap, a 1024-token total output budget per turn and at most 15 actions.

ModelMethodValidation-selected stepValidation successTest success
Qwen3-4B-InstructMemoryless GRPO903%2.34%
Qwen3-4B-InstructFull-history GRPO10079%75.78%
Qwen3-4B-InstructCCM + GRPO + distillation10083%86.72%
Qwen3-8BMemoryless GRPO, thinking enabled803%2.34%
Qwen3-8BMemoryless GRPO, thinking disabled502%3.13%
Qwen3-8BFull-history GRPO8084%82.81%
Qwen3-8BCCM + GRPO + distillation6076%75.00%

Q6: What is the total training compute and wall-clock time required for the full-history teacher forward passes during GRPO training?

Using Webshop, we report the full-history teacher-scoring cost over completed CCM + GRPO + Distill runs:

ModelFLOPsTeacher scoring timeTotal training timeTeacher time share
Qwen3-4B-Instruct3.13 x 10^181h28m00s10h47m51s13.6%
Qwen3-8B5.96 x 10^182h18m47s24h34m54s9.41%

Thank you for the additional experiments and clarifications. They address some of my concerns, but my main concern about the practical benefit of CCM remains unresolved.

The distinction I care about is between reducing retained context and improving end-to-end efficiency. Your response explicitly states that CCM has higher task-attempt completion latency and higher net training time in the measured WebShop 8B workload. Lower live KV occupancy and lower TTFT therefore do not establish faster task completion or more efficient RL training. Similarly, higher output-token throughput is not sufficient evidence of useful throughput when the method itself generates substantially more text.

The cache accounting in Appendix A makes this tradeoff particularly relevant. The reported cache-read share decreases from 90.7% to 35.6% for Sonnet, from 88.3% to 0.0% for Opus, and from 92.6% to 42.3% for Kimi. Thus, the reduction in context length comes with substantially reduced cache reuse in these configurations. What remains unclear is whether the memory savings compensate for this loss of reuse and the additional generation overhead under a practical deployment or training budget.

One plausible application is a memory-constrained setting in which smaller active contexts permit more concurrent trajectories. However, increased concurrency does not necessarily translate into higher end-to-end throughput. In my view, retaining full history while making effective use of prefix caching, tuning concurrency to the available memory, or using lower-precision KV caching such as FP8 where supported are more conventional ways to address this constraint. I suspect these approaches may offer a better practical tradeoff than repeatedly rewriting memory, especially given the reduced cache reuse and additional generation reported here. The current evidence has not convinced me that CCM offers a practical advantage over these alternatives.

For inference, useful measurements would include completed and successfully solved tasks per unit time, together with success rate. For RL training, I would like to see wall-clock time or GPU-hours required to reach a comparable held-out success rate, including the full-history teacher-scoring overhead. Lower memory occupancy alone does not establish a training-efficiency advantage.

I also acknowledge that adding distillation improves CCM over CCM trained with GRPO alone, particularly on WebShop. However, this does not establish that CCM is preferable to a trained full-history agent. In the supplied manuscript, CCM with distillation exceeds full-history GRPO on WebShop 4B, but falls below it on WebShop 8B and Endless Terminals. Your new MEM1-style comparison is likewise mixed across model sizes.

My remaining concern is therefore not whether CCM can reduce retained context, which the experiments demonstrate, or whether distillation can improve CCM, which the WebShop results support. It is whether the proposed approach offers a demonstrated practical advantage over competitive alternatives. Please identify the operating regime in which it does, and evaluate that regime directly. The current evidence does not yet establish an end-to-end throughput or RL training-efficiency benefit.

Overall, I recognize the contribution of the proposed training loss, particularly its improvement over CCM trained with GRPO alone. However, these gains do not translate into a consistent improvement over full-history GRPO. I remain unconvinced about the practical application of the overall approach. The reported increase in net training time, together with reduced cache reuse and additional generation overhead, suggests that CCM may worsen rather than improve end-to-end RL training throughput in the evaluated setting. Although this does not establish that CCM is unhelpful in every setting, the paper has not demonstrated a practical benefit that offsets these costs. I therefore do not find the demonstrated contribution sufficient to justify acceptance.

Thank you for clarifying the distinction between reduced retained context and end-to-end efficiency. We agree that lower KV occupancy, lower time to first token, or higher output-token throughput alone do not establish faster task completion or more efficient RL training. Our measured 8B training workload does not demonstrate a training-efficiency advantage.

Direct Inference Measurements

We have now completed inference profiling of the validation-selected WebShop 4B checkpoint, measuring completed attempts and successfully solved tasks per unit time rather than token throughput alone.

MetricFull-history GRPOCCM + GRPO + Distill
Test success78.13% (100/128)82.81% (106/128)
Collection wall time449.43 s287.24 s
Mean task-attempt latency9.46 s7.68 s
Completed attempts/hour1,025.31,604.2
Successfully solved tasks/hour801.01,328.5
Mean time to first token57.02 ms43.07 ms
Mean live KV/GPU0.3405 GiB0.2716 GiB
Observed peak live KV/GPU1.3601 GiB0.6350 GiB
Total generated tokens, including memory and actions39,88356,783
Inference TFLOPs/task72.9830.74

In this configuration, CCM generates more output tokens but nevertheless completes the task set faster, with 65.9% higher successful-task throughput, 36.1% lower collection time, and 53.3% lower observed peak live KV occupancy.

Additional training cost could potentially be amortized through repeated inference if serving costs are lower as shown here.

Context Management And Systems Optimizations

We agree that cache-friendly prompt layouts, concurrency tuning, and lower-precision KV caching are important optimizations. Our work addresses a complementary question: can the agent learn what information to retain and how to update it as new observations arrive? Systems optimizations improve how a given context is stored and processed; model-controlled context management additionally changes what that context contains. By maintaining an explicit, inspectable memory, this approach leverages the agent’s own ability to organize task-relevant information, with the goal of improving long-horizon efficiency and performance.

Model-controlled context management is an active research direction, with related work reporting benefits on long-horizon agent tasks. Examples include optimized observation/history compression in ACON, joint training of task execution and summary generation in CompactionRL, and autonomous context editing and retrieval in ACM. These findings support its potential practical value.

Cache Reuse

We acknowledge that rewriting context can reduce cache reuse and increase generation costs. Opus's particularly large cache-read drop may also reflect Bedrock's higher minimum cacheable prefix: 4,096 tokens for Opus 4.6 versus 1,024 for Sonnet 4.6. This may help explain Opus's zero cache-read share, but does not remove the observed cache-reuse tradeoff.

Reviewer_eFDGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies Continuous Context Management (CCM), where an agent carries forward model-written memory and the newest observation instead of its full interaction history. It evaluates this protocol without training and introduces a training recipe combining GRPO with gated action-token supervision from a frozen initial model conditioned on full history.

Critical strengths

S1. The caching-aware cost breakdown is useful. The evaluation distinguishes shorter prompts from lower API cost and reports cases where compression makes the agent less successful or more expensive.

S2. Adding the auxiliary loss improves the reported WebShop results over CCM + GRPO. Distillation alone solves only 0/128 and 1/128 tasks for the 4B and 8B models, respectively, indicating that it does not replace GRPO in this setting.

Critical weaknesses

W1. The protocol's methodological novelty is limited. The per-turn memory protocol closely follows MEM1, as acknowledged by the authors, and the paper does not compare it with periodic compaction at a comparable context budget.

W2. The benefit attributable to privileged history is not isolated experimentally. The training recipe adapts SDAR, but its teacher–student confidence gap reflects differences in both context and model parameters. My main concern is that the existing ablations do not isolate the contribution of full history from a more general auxiliary training effect. The paper does not report a control using the same frozen teacher and auxiliary objective with compact context instead of full history.

Questions for authors

Q1. How does success change if memory is consolidated periodically rather than every turn, at a comparable context budget?

Q2. Could you compare full-history and compact-context supervision using the same frozen teacher and auxiliary objective to isolate the contribution of privileged history?

Q3. Does the auxiliary loss still provide gains when added to CCM + GiGPO?

3 comments

Thank you for the thoughtful feedback. We respond to your concerns below.

Q1: How does success change if memory is consolidated periodically rather than every turn, at a comparable context budget?

We added a periodic-compaction control on WebShop using Qwen3-4B-Instruct. All conditions use the same per-call limits of 4,096 input tokens and 1,024 output tokens, with at most 15 environment actions. Checkpoints are selected on the separate 100-task validation set, and results are measured on the fixed 128-task test set using evaluation seed 42.

For periodic compaction, the agent generates a summary after every four actions while the episode remains active. The summary replaces the preceding interaction history. Subsequent prompts contain the original task, the latest summary, the current observation, and the actions and observations accumulated since the last compaction.

Qwen3-4B-Instruct

MethodValidation-selected stepTest successMean memory tokensMaximum memory tokensMedian peak prompt tokensMaximum prompt tokens
Full-history GRPO10075.78%----1757.54096
CCM-GRPO9021.09%54.56710461276
CCM-GRPO + Distill10086.72%60.1891041.51279
Periodic compaction, every four actions5084.38%208.210241547.53720

Periodic compaction is competitive with CCM-GRPO + distillation on task success but uses larger peak prompts in these evaluations.

Q2: Could you compare full-history and compact-context supervision using the same teacher and auxiliary objective to isolate the contribution of privileged history?

We completed a compact-context teacher control using Qwen3-4B-Instruct on WebShop.

We select checkpoints using the 100-task validation set and report success on the separate 128-task test set, using evaluation seed 42.

Teacher conditioning for CCM-GRPO + DistillValidation-selected stepTest success
Same compact context as the student8059.38%
Full interaction history10086.72%

The full-history condition scores 27.34 percentage points higher in this comparison, consistent with privileged history contributing beyond the compact-context auxiliary objective.

Q3: Does the auxiliary loss still provide gains when added to CCM + GiGPO?

We trained Qwen3-4B-Instruct on WebShop using CCM + GiGPO, with and without the auxiliary objective. It was evaluated on the same fixed 128-task test set, with thinking disabled, 4096 input / 1024 output tokens, 15 actions, temperature 0.4 and seed 42.

MethodCheckpointTest success
CCM + GiGPO10083.59% (107/128)
CCM + GiGPO + Distill10012.50% (16/128)
CCM + GRPO + Distill, our method10086.72% (111/128)

Our method achieves 86.72% under the same protocol. The auxiliary objective did not improve final-checkpoint performance when combined with GiGPO in this run.

We have not completed the corresponding Endless Terminals GiGPO comparison due to compute constraints.

Separately, Li et al., Credit Where It Matters (https://arxiv.org/html/2610.03634v1#S1) report only four recurring anchor states out of 25 in two traced terminal tasks under their GiGPO implementation, suggesting that sparse state recurrence can limit exact-state grouping. This observation motivates caution when extending GiGPO to terminal environments.

Thanks for the clarification, most of my concerns are resolved, I raise my score to 3.

We sincerely thank you for your positive reassessment and increased score.

Reviewer_NtFuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Continuous Context Management (CCM), where an agent updates a memory at each turn and uses it together with the task and latest observation instead of retaining the full transcript. It evaluates CCM on TerminalBench-2, then trains CCM agents on WebShop and Endless Terminals using GRPO and an auxiliary distillation loss. A frozen copy of the initial model sees the full history and scores actions sampled by the student.

Critical strengths

S1. The paper addresses a practical problem for multi-turn agents: retaining useful information without allowing the prompt to grow throughout an episode.

S2. The cost analysis accounts for prompt caching and reports cases where CCM reduces input tokens but increases API cost. This makes the tradeoffs easier to assess.

S3. The auxiliary loss substantially improves WebShop success over CCM + GRPO. The paper also provides detailed prompts, hyperparameters, and token accounting.

Critical weaknesses

W1. It is unclear whether the gains come from full-history information or policy regularization. Keeping the teacher and loss unchanged while giving the teacher compact context, alongside tuning reference KL, would help distinguish these explanations.

W2. Selecting the best checkpoint on the test set may inflate reported performance. Checkpoints should be selected on validation data, with results reported across multiple independent training seeds.

W3. Without periodic compaction, observation masking, and memoryless baselines, it is unclear whether updating memory at every turn is necessary or preferable.

W4. The efficiency claims rely on token counts rather than direct measurements. Peak KV-cache memory, task latency, throughput, and the training overhead of teacher scoring should be reported.

W5. The 4,096-token input limit may disadvantage the WebShop full-history baseline. Please report truncation frequency and evaluate this baseline with a larger context window.

W6. More frequent memory updates do not necessarily indicate more useful memory. Removing or replacing memory at evaluation time would help establish whether the trained agent relies on retained information.

W7. Generating memory before an action also gives the model extra computation. A comparable temporary-text baseline, without cross-turn retention, would help separate this benefit from persistent memory.

Questions for authors

Q1. Does the auxiliary loss still help when the same frozen teacher receives compact context rather than full history? How does it compare with tuning the reference-KL coefficient?

Q2. Can you report results with validation-selected checkpoints and multiple independent training seeds, together with uncertainty estimates?

Q3. How does CCM compare with periodic compaction and observation masking at similar context budgets? How well does a memoryless agent trained with GRPO perform?

Q4. What are the measured peak KV-cache memory, task-completion latency, and throughput? How much training time does teacher scoring add?

Q5. How often is the WebShop full-history baseline truncated, and does increasing its context limit change the comparison?

Q6. How does the trained policy perform when its memory is removed or replaced at evaluation time, particularly on tasks requiring information from earlier turns?

Q7. How does CCM compare with a policy that generates a temporary text block before each action but does not carry it into the next turn, under a comparable generation budget?

5 comments

Thank you for the thoughtful feedback. We respond to your concerns below.

Q1: Does the auxiliary loss still help when the same frozen teacher receives compact context rather than full history? How does it compare with tuning the reference-KL coefficient?

We added two controls using Qwen3-4B-Instruct: (i) the same auxiliary objective with the teacher receiving the student's compact context rather than full history, and (ii) CCM-GRPO with the reference-KL coefficient increased from 0.01 to 0.1 (10x).

For all methods, checkpoints are selected by success on the 100-task validation set. We report success on the separate 128-task test set below.

MethodTest success
CCM-GRPO, reference KL = 0.0121.09%
CCM-GRPO, stronger reference KL = 0.1 (new)27.34%
CCM-GRPO + Distill, same compact teacher context (new)59.38%
CCM-GRPO + Distill, full-history teacher context86.72%

Q2: Can you report results with validation-selected checkpoints and multiple independent training seeds, together with uncertainty estimates?

We now report results using validation-selected checkpoints rather than selecting the checkpoint with the highest test success. For each method, we select the checkpoint with the highest success on the separate 100-task validation set, breaking ties in favor of the earlier checkpoint. We then report test success from that same checkpoint. The test sets contain 128 tasks for WebShop and 300 tasks for Endless Terminals.

Benchmark and modelMethodSelected stepValidation successTest success
WebShop, Qwen3-4B-InstructFull-history GRPO10079.0%75.78%
WebShop, Qwen3-4B-InstructCCM-GRPO9032.0%21.09%
WebShop, Qwen3-4B-InstructCCM-GRPO + Distill10083.0%86.72%
WebShop, Qwen3-8BFull-history GRPO8084.0%82.81%
WebShop, Qwen3-8BCCM-GRPO5032.0%25.00%
WebShop, Qwen3-8BCCM-GRPO + Distill6076.0%75.00%
Endless Terminals, Qwen3-8BFull-history GRPO3041.0%35.33%
Endless Terminals, Qwen3-8BCCM-GRPO3031.0%29.00%
Endless Terminals, Qwen3-8BCCM-GRPO + Distill5037.0%32.00%

These results address the checkpoint-selection concern. Due to compute restraints, we were not able to do multiple training runs.

Q3: How does CCM compare with periodic compaction and observation masking at similar context budgets? How well does a memoryless agent trained with GRPO perform?

We added periodic-compaction, memoryless-GRPO, and observation masking controls on WebShop. All conditions use the same per-call limits of 4096 input tokens and 1024 output tokens, with at most 15 turns. Checkpoints are selected on the 100-task validation set, and all results below are measured on the separate 128-task test set using evaluation seed 42.

For periodic compaction, the agent generates a summary after every four actions. The summary replaces the preceding interaction history. Subsequent prompts contain the original task, the latest summary, the current observation, and the actions and observations accumulated since the last compaction.

Qwen3 4B Instruct

MethodValidation-selected stepTest successMean memory tokensMaximum memory tokensMedian peak prompt tokensMaximum prompt tokens
Full-history GRPO10075.78%----1757.54096
CCM-GRPO9021.09%54.56710461276
CCM-GRPO + Distill10086.72%60.1891041.51279
Periodic compaction, every four actions5084.38%208.210241547.53720
Memoryless GRPO902.34%----832.51090
Observation-masking GRPO10078.13%----909.51276

Periodic compaction is competitive with CCM-GRPO + distillation on task success but uses larger peak prompts in these evaluations. Observation masking achieves smaller peak prompts than CCM-GRPO + distillation, but lower success.

Memoryless GRPO Across Model Settings

The memoryless agent receives the original task, current observation, and available actions, without prior actions, prior observations, or persistent generated memory.

Model and decoding modeValidation-selected stepTest success
Qwen3-4B-Instruct, non-thinking902.34%
Qwen3-8B, thinking on802.34%
Qwen3-8B, thinking off503.13%

Observation-Masking GRPO

We also completed the observation-masking control for Qwen3-4B-Instruct. Unlike the memoryless agent, it retains past parsed actions but masks all past observations. The original task, full current observation and available actions remain visible; it does not carry generated memory.

MethodValidation-selected stepValidation successTest success
Observation-masking GRPO, Qwen3-4B-Instruct10080%78.13%

We selected the checkpoint with the best validation success (step 100), which achieved 78.13% test success. Observation masking substantially outperforms the 4B memoryless baseline (2.34%), but remains 8.59 percentage points below CCM-GRPO + distillation (86.72%) in these single-run evaluations.

Q5: How often is the WebShop full-history baseline truncated, and does increasing its context limit change the comparison?

We audited the full-history prompts in the fixed 128-task test evaluations. We report both the fraction of turns truncated and the fraction of episodes with at least one truncated turn.

Model and conditionTruncated turnsEpisodes with truncation
Qwen3-4B-Instruct, untrained full history262/1397 (18.75%)74/128 (57.81%)
Qwen3-8B, untrained full history92/904 (10.18%)29/128 (22.66%)

We evaluated the Webshop full-history baseline with the input cap increased from 4,096 to 32,768 tokens, keeping the fixed 128-task test set, 1024-token output budget, 15-action limit, temperature 0.4 and seed 42 unchanged.

Untrained full-history modelTest success, 4096-token input capTest success, 32,768-token input cap
Qwen3-4B-Instruct3.91%3.91%
Qwen3-8B16.41%15.63%

The larger input limit eliminated prompt truncation in both controls but did not improve their observed success.

Q6: How does the trained policy perform when its memory is removed or replaced at evaluation time, particularly on tasks requiring information from earlier turns?

We do not yet have results for the requested evaluation-time memory-removal or replacement experiment, but we are actively working to complete it and will report the results when available.

As a related control, we trained memoryless agents with GRPO, providing only the original task, current observation and available actions, without prior interaction history or persistent memory. Their validation-selected test success was 2.34% for Qwen3-4B-Instruct, 2.34% for Qwen3-8B with thinking enabled, and 3.13% with thinking disabled. The thinking-enabled condition permits temporary reasoning before each action without carrying it into subsequent turns, testing whether a reasoning space can substitute for persistent memory.This control removes memory during both training and evaluation.

Q7: How does CCM compare with a policy that generates a temporary text block before each action but does not carry it into the next turn, under a comparable generation budget?

We evaluated this experiment by training Qwen3-8B memoryless GRPO with native thinking enabled, and a corresponding thinking-disabled control. In the thinking-enabled condition, the policy generates reasoning before its action, but neither that reasoning nor earlier actions/observations are carried into the next turn. Each new prompt contains only the original task, current observation and available actions.

All conditions below have a 4096-token input cap, a 1024-token total generation cap per turn, and a 15-action limit. We select checkpoints by validation success and report the separate 128-task test success using evaluation seed 42.

Qwen3-8B methodValidation-selected stepTest success
Memoryless GRPO, thinking disabled503.13%
Memoryless GRPO, temporary thinking enabled802.34%
CCM-GRPO, persistent compact memory5025.00%
CCM-GRPO + Distill, persistent compact memory6075.00%

Under this evaluation protocol, enabling temporary reasoning without carrying memory does not close the gap to CCM. The distinction is therefore not simply whether the model generates text before an action, but whether generated information remains available on later turns.

Reviewer_4ayhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Continuous Context Management (CCM), where agents maintain compact, updated memories throughout a task. By combining reinforcement learning with full-history distillation, the method improves agents’ ability to act using limited context. Experiments show substantial reductions in context usage and strong gains on some shopping tasks, while improvements on terminal tasks are smaller. CCM offers a promising approach to memory-efficient agents, though it does not consistently outperform full-history methods.

Critical strengths
  1. Well-structured training design. Combining episode-level GRPO with action-token guidance from a full-history teacher addresses the limited feedback available for learning from compact memory. The teacher scores the student’s own trajectories, requiring neither additional teacher rollouts nor successful reference solutions.

  2. Meaningful empirical performance. On WebShop, the 4B model outperforms full-history GRPO while using substantially shorter prompts, demonstrating a useful accuracy–context tradeoff. The GRPO-only and distillation-only ablations support the value of combining the two training signals.

Critical weaknesses
  1. Unclear advance over existing methods. The per-turn memory consolidation closely follows MEM1, while the distillation objective adapts SDAR’s confidence-gap gating. Also, without matched comparisons against existing memory-management methods or periodic compaction, I am not sure whether the experiments have established a clear advantage over these baseline alternatives.

  2. Uncertain robustness of the gains. Best checkpoints are selected on the same fixed sets used to report performance, without uncertainty estimates across training runs. This particularly weakens the modest improvement on Endless Terminals, where the advantage reverses at the final checkpoint.

Questions for authors

See the weaknesses above

3 comments

Thank you for the thoughtful feedback. We respond to your concerns below.

Q1: Unclear advance over existing methods. The per-turn memory consolidation closely follows MEM1, while the distillation objective adapts Distill's confidence-gap gating. Also, without matched comparisons against existing memory-management methods or periodic compaction, I am not sure whether the experiments have established a clear advantage over these baseline alternatives.

We agree that per-turn memory consolidation and confidence-gap gating are not individually new, and do not claim to have introduced either component. Our contribution is their adaptation to compact-context agent training: the student acts using compact context, while an auxiliary objective scores its sampled actions using privileged full interaction history available during training. We now provide controls examining the contributions of teacher conditioning, the auxiliary objective and stronger reference-KL regularization, alongside MEM1-style PPO, observation-masking, memoryless-GRPO and periodic-compaction baselines.

Teacher Context And Stronger KL Controls

All rows below use Qwen3-4B-Instruct on WebShop. We select the checkpoint with the highest success on the separate 100-task validation set, break ties in favor of the earlier checkpoint, and report test success from that same checkpoint on the fixed 128-task test set.

MethodValidation-selected stepValidation successTest success
CCM-GRPO, reference KL = 0.019032%21.09%
CCM-GRPO, reference KL = 0.18032%27.34%
CCM-GRPO + Distill, compact-context teacher8063%59.38%
CCM-GRPO + Distill, full-history teacher10083%86.72%

The compact-context teacher control retains the same auxiliary objective but gives the teacher the student’s context instead of full history. Its test success is 27.34 percentage points below the full-history teacher condition, supporting a contribution from privileged history in this run. Increasing the reference-KL coefficient from 0.01 to 0.1 improves CCM-GRPO test success from 21.09% to 27.34%, but does not reproduce the 86.72% achieved by CCM + distillation with full-history supervision.

MEM1-Style PPO And Context Baselines

We agree that MEM1 is closely related. Both approaches maintain a compact state instead of retaining the entire interaction history. MEM1 uses PPO with a learned critic, while our method uses critic-free GRPO plus full-history auxiliary supervision. MEM1's implementation uses custom full-trajectory attention masks for the changing context. Our adaptation reconstructs each turn's retained context for use with standard trainers.

Periodic compaction uses the original task and recent full history, generating a summary after every four actions.

The following tables report both success and context behavior. All trained checkpoints are validation-selected. WebShop evaluation uses seed 42, temperature 0.4, a 4096-token input cap, a 1024-token output cap per call and at most 15 actions.

Qwen3-4B-Instruct

ConditionSelected stepTest successMemory update (%)Prefix retention (%)Mean memory tokensMaximum memory tokensMedian peak prompt tokensMaximum prompt tokens
Untrained full history--3.91%--------40964096
Full-history GRPO10075.78%--------1757.54096
Untrained CCM--0.78%57.669.6225.78611232.51908
CCM-GRPO9021.09%0.2100.054.56710461276
CCM-GRPO + Distill10086.72%100.077.260.1891041.51279
MEM1-style PPO10078.13%99.22.233.510913801643
Periodic compaction, every four actions5084.38%16.798.5208.210241547.53720
Memoryless GRPO902.34%--------832.51090
Observation-masking GRPO10078.13%--------909.51276

Qwen3-8B

ConditionSelected stepTest successMemory update (%)Prefix retention (%)Mean memory tokensMaximum memory tokensMedian peak prompt tokensMaximum prompt tokens
Untrained full history--16.41%--------2258.54096
Full-history GRPO8082.81%--------17294096
Untrained CCM--0.00%15.096.7511.810131582.52127
CCM-GRPO5025.00%10.799.7544.39161282.51881
CCM-GRPO + Distill6075.00%92.383.1194.53371141.51471
MEM1-style PPO10080.47%99.32.3117.34481429.51758
Memoryless GRPO, thinking enabled802.34%--------9351152
Memoryless GRPO, thinking disabled503.13%--------8701094

On 4B, CCM-GRPO + Distill scores 8.59 percentage points above MEM1-style PPO and 2.34 points above periodic compaction. On 8B, MEM1-style PPO scores 5.47 points above our method. MEM1-style PPO uses shorter mean summaries, while CCM-GRPO + Distill has smaller median and maximum peak prompts in both model sizes. Periodic compaction is a strong success baseline but has larger peak prompts than CCM-GRPO + Distill in the 4B comparison.

Auxiliary Objective

We also tested whether the auxiliary objective improves CCM when combined with GiGPO rather than GRPO. We trained Qwen3-4B-Instruct on WebShop with CCM + GiGPO, with and without the auxiliary objective. They were evaluated on the same fixed 128-task test set, with thinking disabled, a 4096-token input cap, 1024 output tokens per turn, at most 15 actions, temperature 0.4 and seed 42.

MethodCheckpointTest success
CCM + GiGPO10083.59% (107/128)
CCM + GiGPO + Distill10012.50% (16/128)
CCM + GRPO + Distill, our method10086.72% (111/128)

Our method achieves 86.72% under the same protocol. The auxiliary objective did not improve final-checkpoint performance when combined with GiGPO in this run.

We have not completed the corresponding Endless Terminals GiGPO comparison due to compute constraints.

Separately, Li et al., Credit Where It Matters (https://arxiv.org/html/2610.03634v1#S1) report only four recurring anchor states out of 25 in two traced terminal tasks under their GiGPO implementation, suggesting that sparse state recurrence can limit exact-state grouping. This observation motivates caution when extending GiGPO to terminal environments.

Q2: Uncertain robustness of the gains. Best checkpoints are selected on the same fixed sets used to report performance, without uncertainty estimates across training runs. This particularly weakens the modest improvement on Endless Terminals, where the advantage reverses at the final checkpoint.

Checkpoints are now selected exclusively by success on the separate 100-task validation set, with ties resolved in favor of the earlier checkpoint, and test success is reported from that same checkpoint. The validation and test sets are fixed and separate; WebShop test evaluation uses 128 tasks and Endless Terminals uses 300 tasks.

Benchmark and modelMethodSelected stepValidation successTest success
WebShop, Qwen3-4B-InstructFull-history GRPO10079%75.78%
WebShop, Qwen3-4B-InstructCCM-GRPO9032%21.09%
WebShop, Qwen3-4B-InstructCCM-GRPO + Distill10083%86.72%
WebShop, Qwen3-8BFull-history GRPO8084%82.81%
WebShop, Qwen3-8BCCM-GRPO5032%25.00%
WebShop, Qwen3-8BCCM-GRPO + Distill6076%75.00%
Endless Terminals, Qwen3-8BFull-history GRPO3041%35.33%
Endless Terminals, Qwen3-8BCCM-GRPO3031%29.00%
Endless Terminals, Qwen3-8BCCM-GRPO + Distill5037%32.00%

Due to compute constraints, we could not run multiple independent training seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.