acceptodds
Under review as a conference paper at ICLR 2027

RECAP: Training Long-horizon LLM Agents with Context Compaction

Abstract

Context compaction enables large language model (LLM) agents to tackle long-horizon tasks within limited context budgets by replacing interaction histories with compressed summaries. However, learning effective compaction jointly with the agent policy remains challenging, as each summary’s effect on task success unfolds over many subsequent interactions. We introduce RECAP, a reinforcement learning (RL) framework based on the principle that a summary should preserve the behavior induced by the original interaction history. We formalize this principle with a behavior distortion objective that measures the divergence between the agent’s subsequent behavior under compressed and original contexts. Its gradient decomposes into two complementary learning signals: distortion-based reward for summary generation and self-distillation for subsequent actions. Together with the task-success objective, these signals jointly optimize context compaction and the agent policy. Experiments on BrowseComp-Plus and SWE-bench Verified demonstrate the benefit of \method compared to other RL baselines.

Primary area:
reinforcement learning
Keywords:
LLM Agents, Reinforcement Learning, Context Compaction, Long-horizon Agents, Self-distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jtFtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes RECAP, an RL framework for jointly training context compaction and agent policies in long-horizon LLM agents. The central idea is that an effective summary should preserve the agent's behavior induced by the original full history. The authors design a behavior distortion objective based on the KL divergence between the policies conditioned on the original full history and compacted contexts.

Experiments on BrowseComp-Plus and SWE-bench demonstrate improvements from RECAP.

Critical strengths
  1. Context compaction has been demonstrated useful in long-horizon task. The paper aims to simultaneously optimize the context compaction and agent policy. The paper has clear motivation.

  2. The experiments demonstrate improvements across both deep-research and software-engineering tasks.

  3. The paper is well written and easy to follow.

Critical weaknesses
  1. My main concern is the assumption that the full-context policy provides the desired behavior that the compacted policy should imitate. Context compaction is not necessarily just an information-preservation mechanism. In long-horizon agent tasks, the full interaction history may contain redundant observations. By removing such distracting information and highlighting task-relevant states, a compacted context may actually enable the same model to make better decisions than when conditioned on the full context.

In other words, the proposed self-distillation objective encourages behavior preservation, whereas the ultimate goal of context management should be improving task performance, which does not always require preserving the original behavior. The paper does not sufficiently investigate when these two objectives are aligned or conflicting.

  1. The experimental results show that self-distillation improves overall performance, which supports the effectiveness of the proposed method. However, these results do not establish that imitating the full-context policy is always beneficial. In particular, it remains unclear what happens when the compacted-context policy already outperforms the full-context policy at a given decision state. Does self-distillation still improve subsequent decisions, or does it pull the compacted policy toward an inferior behavior?
Questions for authors

See weaknesses.

5 comments

We thank the reviewer for the invaluable comments. We’re happy that the reviewer thinks our paper is clearly motivated, well written and easy to follow. We address the reviewer’s concern below.


W1: My main concern is the assumption that the full-context policy provides the desired behavior that the compacted policy should imitate. The ultimate goal of context management should be improving task performance, which does not always require preserving the original behavior.

We thank the reviewer for the important question. RECAP uses the full-context policy as a self-teacher for training the compacted policy of the agent. To test whether the full-context policy serves as a good teacher, we compare the performance of full-context and compacted policies in different scenarios, as shown below.

Qwen3.5-4BQwen3.5-9BQwen3-CoderQwen3.5-4B Trained
Full-context policy49.253.024.761.8
Compacted policy40.747.025.062.8

Table 1: Task success rate (%) on BrosweComp-Plus.

Qwen3.5-4BQwen3.5-9BQwen3-CoderQwen3.5-4B Trained
Full-context policy49.856.843.754.3
Compacted policy45.852.344.151.1

Table 2: Task success rate (%) on SWE-bench Verified.

The full-context policy performs better in 5 out of 8 settings, with an average advantage of 3.1 points. For Qwen3-Coder, task success rates were similar across the two settings, as approximately 90% of trajectories terminated without triggering context compaction. The only exception is the trained Qwen3.5-4B on BrowseComp-Plus (-1.0%), where the difference is minor. The results strongly indicate that the full-context policy generally provides a strong reference for the compacted policy.

Finally, we appreciate the reviewer’s insight that context management can remove redundant observations while preserving task-relevant information. RECAP is compatible with this form of selective compression. Redundant information can be discarded without penalty as long as its removal does not affect subsequent behavior, whereas omitting information that matters for future decisions increases behavioral distortion and is penalized. Thus, the distortion objective provides a functional criterion for what should be preserved and what can be safely removed. Importantly, behavior preservation remains an auxiliary regularizer alongside the task reward, allowing task optimization to favor compacted behaviors that differ from the full-context behavior and potentially achieve better task performance. We’ll include the discussion in the revised manuscript.

W2: It remains unclear what happens when the compacted-context policy already outperforms the full-context policy at a given decision state. Does self-distillation still improve subsequent decisions, or does it pull the compacted policy toward an inferior behavior?

In response to the reviewer’s suggestion, we take the trained Qwen3.5-4B whose compacted policy behaves slightly better than the full-context policy (62.8% vs. 61.8% in the table above). We continue training for 10 steps using a frozen full-context teacher without task-success rewards. We separately optimize the summary reward (gsumg_{sum}), post-compaction action distillation (gpostg_{post}), or both. The results are summarized below:

Accuracy (%)
Checkpoint62.8
+gsumg_{sum}63.7
+gpostg_{post}60.3
+Both60.7

*Table 3: Task performance after additional training using behavior distortion. *

Interestingly, the summary reward gsumg_{sum} alone improves the performance, while self-distillation on actions with gpostg_{post} reduces the task accuracy. These results show that behavior preservation alone does not always improve the task performance, and support the design choice of RECAP which takes the distortion objective as an auxiliary regularization objective alongside the task success reward. We will clarify this in the revised manuscript.

Many thanks to authors' detailed response. I still have two questions:

  1. What is the detailed setting for Compacted policy?

  2. Have you evaluated task success rates across different task lengths (e.g., different numbers of turns)? Does the full-context policy consistently outperform the compacted policy on longer tasks?

Thank you for your the kind follow-up questions:

  • Setting for compacted policy: We use 32K context window with context compaction. When the active context reaches 32k token limit, the model generates a handoff summary using the prompt specified in Appendix E.1. The agent then resumes the same task with the original task instructions, the summary, and the most recent action and its observation (k=1k=1 on both benchmarks). We allow up to three compactions per task.

  • Task success rates across different task lengths: We thank the reviewer for the great suggestion. We group the tasks by their average number of steps and compare the performance of full-context policy and compacted policy below

TaskModelAvg. stepsFull context (%)Compacted (%)  Gap (pp)
BrowseComp-PlusQwen3.5-4B<10<1086.982.1 +4.8
10−2010 - 2092.484.8 +7.6
≥20\geq 2032.322.9 +9.4
BrowseComp-PlusQwen3.5-4B Trained<10<1091.994.2 −2.3
10−2010 - 2082.985.5 −2.6
≥20\geq 2031.531.2 +0.3
SWE-bench VerifiedQwen3.5-4B<100<10060.856.5  +4.3
100−150100 - 15050.545.3 +5.2
≥150\geq 15040.733.8 +6.9
SWE-bench VerifiedQwen3.5-4B Trained<100<10064.464.0 +0.4
100−150100 - 15048.347.2 +1.1
≥150\geq 15040.034.8 +5.2

Both policies generally achieve lower success rates on longer tasks. For the base model, the full-context policy consistently outperforms the compacted policy, with the performance gap increasing as task length increases on both environments. After training, the gap reduces for BrowseComp-plus, while still persists for SWE-bench Verified. Overalll, these results suggest that the advantage of full-context policy is more pronounced on the challenging long-horizon tasks.

Reviewer_tWjFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work investigates the problem of training long-horizon LLM agents with context compaction, where the agent replaces its long interaction history with a summary to stay within a limited context window. It identifies that the final task reward gives only weak and noisy feedback for learning good summaries, because the effect of a single summary spreads over many later steps. To address this, it proposes RECAP, which measures "behavior distortion": the token-level KL divergence between the agent's actions under the summary and under the full history, with the same model acting as its own teacher. This yields two extra training signals: a reward for summaries that cause less distortion, and self-distillation for the actions after compaction. With Qwen3.5-4B and a 32K context, RECAP improves accuracy over task-reward-only RL by 4.6% on BrowseComp-Plus and 4.3% on SWE-bench Verified, while using fewer tool calls and tokens.

Critical strengths
  1. This manuscript formulates summary compaction as a behavior distortion problem, measuring summary quality by the KL divergence between the same model's action distributions under the full and compressed contexts, which provides direct and dense supervision for both summary generation and post-compaction actions without extra rollouts or a separate teacher model, easing the credit assignment problem of relying only on the final task reward.

  2. The method is theoretically well grounded. Starting from a single behavior distortion objective, the authors decompose its gradient and naturally obtain two complementary learning signals: a distortion-based policy-gradient reward for summary generation, and a token-level self-distillation signal for post-compaction actions. This is a principled derivation rather than a heuristic combination of existing techniques. The appendix is also transparent about the approximation involved, namely a surrogate objective that stops gradients through continuation sampling (equivalent to a per-token estimator with γ = 0), which makes the underlying assumptions and potential bias explicit.

Critical weaknesses
  1. The experimental comparison is limited, as the only training baseline is task-reward-only GRPO. The paper does not compare against related context management methods (e.g., CompactionRL, Context-Folding, MEM1), simple baselines such as sliding-window truncation, or methods designed to improve credit assignment (e.g., GiGPO, turn-level rewards).

  2. The core assumption that lower behavior distortion leads to better task performance is not verified. Recent work shows that self-distillation from a teacher with privileged information can optimize a signal decoupled from task success [1], and can even hurt multi-turn agents, as the student learns to act as if it had information it never observed [2]. In RECAP, when the summary loses information, the right action for the student (e.g., searching again) may differ from the teacher's, yet RECAP penalizes this deviation. Moreover, full-context behavior is not always the right target, since longer contexts can degrade performance. An analysis of the correlation between distortion and task success would help.

[1] Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation

[2] From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents

Questions for authors

see weakness

2 comments

We thank the reviewer for the constructive comment and overall positive evaluation of our work. We’re encouraged that the reviewer finds our method theoretically well grounded and appreciates its principled formulation. We address the reviewer’s concerns below.


W1: The experimental comparison is limited, as the only training baseline is task-reward-only GRPO. The paper does not compare against related context management methods (e.g., CompactionRL, Context-Folding, MEM1), simple baselines such as sliding-window truncation, or methods designed to improve credit assignment (e.g., GiGPO, turn-level rewards).

We thank the reviewer for the important question. In response to the reviewer’s suggestion, we first compare the results with two alternative context management approaches (sliding-window and context folding). The results are shown below.

ModelBrowseComp-plusSWE-bench Verified
Sliding WindowQwen3.5-4B43.848.9
Context FoldingQwen3.5-4B33.038.4
Context CompationQwen3.5-4B40.745.8
Qwen3.5-4B+RECAP62.851.1

Table 1: Comparison of RECAP to other context management approaches.

For the base model, sliding-window truncation outperforms standard context compaction by 3.1% on both environments, while Context Folding performs worse than both methods. Nevertheless, RECAP achieves the highest success rate on both benchmarks, outperforming all evaluated context management alternatives.

We next compare RECAP with CompactionRL, a PPO-based training method. Since CompactionRL optimizes the final task reward, its training is complementary to RECAP's behavior distortion objective. We therefore evaluate CompactionRL both with and without RECAP.

Performance
Qwen3.5-4B40.7
+ CompcationRL48.3
+ RECAP (CompactionRL)54.0
+ RECAP (GRPO)62.8

Table 2: Comparison of RECAP and CompactionRL.

The results show that CompactionRL improves accuracy from 40.7% to 48.3%, and incorporating RECAP further increases it to 54.0%. This demonstrates that RECAP's behavior distortion objective can also improve performance under PPO-based training. We will in include these additional baselines in the revised manuscript.

W2.1: The core assumption that lower behavior distortion leads to better task performance is not verified. Moreover, full-context behavior is not always the right target, since longer contexts can degrade performance.

RECAP measures the distortion to the full-context policy to train the agent, under the assumption that full-context policy is a strong teacher for long-horizon tasks. To verify this assumption, we compare the task success rates of full-context and compacted policies across different models and environments. The results are summarized below

Qwen3.5-4BQwen3.5-9BQwen3-CoderQwen3.5-4B Trained
Full-context policy49.253.024.761.8
Compacted policy40.747.025.062.8

Table 3: Task success rate (%) on BrosweComp-Plus.

Qwen3.5-4BQwen3.5-9BQwen3-CoderQwen3.5-4B Trained
Full-context policy49.856.843.754.3
Compacted policy45.852.344.151.1

Table 4: Task success rate (%) on SWE-bench Verified.

The results strongly indicate that the full-context policy generally provides a strong reference for the compacted policy, outperforming it in 5 out of 8 settings, with an average improvement of 3.1% across all tasks and environments. For Qwen3-Coder, task success rates were similar across the two settings because most of the trajectories finishes without triggering compaction. The only exception is the trained Qwen3.5-4B on BrowseComp-Plus (−1.0%). We hypothesize that this is due to the explorative feature of the task, where compaction might help clean the context and incentivize exploration [3, 4]. We’ll include the discussion in the revised manuscript.


W2.2: Recent work shows that self-distillation from a teacher with privileged information can optimize a signal decoupled from task success [1], and can even hurt multi-turn agents, as the student learns to act as if it had information it never observed [2]. In RECAP, when the summary loses information, the right action for the student (e.g., searching again) may differ from the teacher's, yet RECAP penalizes this deviation.

We thank the reviewer for the insightful question. RECAP’s distortion objective contains both reward to the generated summary (gsumg_{sum}) and self-distillation in subsequent actions (gpostg_{post}). Summaries that omit necessary information will receive less credit through gsumg_{sum} during training.

To further examine the effect of behavior distortion to task success, we take the trained 150 checkpoints on BrowseComp-plus, and continue to train it with a frozen teacher for an additional 10 steps. We separately use the summary objective (gsumg_{sum}), post-compaction self-distillation (gpostg_{post}), or both. The results are shown below.

Accuracy (%)
Trained62.8
+gsumg_{sum}63.7
+gpostg_{post}60.3
+Both60.7

Table 5: Task performance after additional training using behavior distortion.

Interestingly, gsumg_{sum} alone improves the performance, while gpostg_{post} yields lower accuracy. These results illustrate that optimizing behavior preservation alone might not improve the task performance, supporting RECAP's design choice that leverages distortion as an auxiliary regularizer alongside task success objectives.

Reference

[1] Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation

[2] From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents

[3] Fork-and-Flush: Escaping Idea Basins in Autoresearch Agents. arXiv preprint arXiv:2610.07447.

[4] Scaling Discovery Requires Forgetting. arXiv preprint arXiv:2610.20563.

Reviewer_bAqxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RECAP, an RL framework for jointly training context compaction and agent behavior. It supplements task rewards with a behavior-distortion objective that compares the same policy under original and compressed contexts. This provides a reward for summary generation and self-distillation signals for subsequent actions. Experiments on BrowseComp-Plus and SWE-bench Verified show improvements over task-reward-only training.

Critical strengths
  1. The method addresses a practical limitation of outcome-only training: summaries receive delayed and indirect supervision. Using their effect on subsequent behavior as an additional learning signal is a simple and reasonable idea.

  2. RECAP improves accuracy by 4.6 and 4.3 percentage points across two different task domains while reducing tool calls and token usage. The ablation studies and experiments across different context budgets further support the effectiveness of the method.

Critical weaknesses
  1. The relationship between behavior preservation and task success is not entirely clear. The original-context policy is not necessarily a better teacher, and matching it could discourage beneficial changes after compaction. Although task rewards are still part of training, I would like to see more analysis of whether the original-context policy is actually a reliable teacher and whether lower distortion leads to better task outcomes. The mixed results from the summary reward alone also raise some questions about this.

  2. The implemented update differs from the stated objective in a few ways. Appendix A.2 uses a surrogate that omits gradients through the continuation distribution, and the implementation normalizes distortion by continuation length rather than using the sum in the formulation. It would be helpful to understand how these approximations affect the results.

  3. The experiments are limited to a single 4B backbone, and the four reported runs reflect evaluation variability rather than independent training variability. Also, the claim of comparable training time is not backed by measurements that account for the additional teacher computation.

Questions for authors

Can you provide a case study comparing RECAP and GRPO summaries and their effects on subsequent actions?

2 comments

We thank the reviewer for the insightful feedback. We’re glad that the reviewer recognizes that our method addresses a practical limitation and finds the approach simple and reasonable. We address the reviewer’s concerns below.


W1: The relationship between behavior preservation and task success is not entirely clear. The original-context policy is not necessarily a better teacher, and matching it could discourage beneficial changes after compaction. I would like to see more analysis of whether the original-context policy is actually a reliable teacher.

We thank the reviewer for raising this important question.

RECAP uses the full-context policy as the teacher to train the agent itself. To examine whether the full-context policy is a better teacher, we compare the task success rates of full-context and compacted policies across different models and environments. The results are summarized below:

Qwen3.5-4BQwen3.5-9BQwen3-CoderQwen3.5-4B Trained
Full-context policy49.253.024.761.8
Compacted policy40.747.025.062.8

Table 1: Task success rate (%) on BrosweComp-Plus.

Qwen3.5-4BQwen3.5-9BQwen3-CoderQwen3.5-4B Trained
Full-context policy49.856.843.754.3
Compacted policy45.852.344.151.1

Table 2: Task success rate (%) on SWE-bench Verified.

The results strongly indicate that the full-context policy generally provides a strong reference for the compacted policy, outperforming it in 5 out of 8 settings, with an average improvement of 3.1% across all models and environments. For Qwen3-Coder, task success rates were similar across the two settings, as approximately 90% of trajectories terminated without triggering context compaction. The only exception is the trained Qwen3.5-4B on BrowseComp-Plus (−1.0%). We hypothesize that this is due to the explorative feature of the task, where compaction might help clean context and incentive exploration [1, 2]. We’ll include these experiments and discussion in the revised manuscript.


W2. The implemented update differs from the stated objective in a few ways. Appendix A.2 uses a surrogate that omits gradients through the continuation distribution, and the implementation normalizes distortion by continuation length rather than using the sum in the formulation. It would be helpful to understand how these approximations affect the results.

We follow the established practice in self-distillation, which treats the prefixes as fixed with stop-gradient [3, 4, 5]. As explained in [5], this corresponds to a token-level discount factor of γ=0\gamma = 0. Prior work also reports no clear benefit from including the gradient terms arising from the continuation distribution [3, 5].

We use length normalization when computing distortion to remove the length bias and balance distortion against task success. To examine its effect, we compare the advantage of behavior distortion objective and task success in the table below

Length normalizationAdvantage ratio
yes0.0154x
no20.9x

Table 3: Ratio of behavior-distortion advantage to task-success advantage.

Without normalization, the compassion advantage is approximately 21 times larger than the task advantage and will dominate the training. Since distortion serves as an auxiliary regularizer, length normalization therefore helps balance its contribution against task success. We will clarify this rationale and the implementation choices in the revised manuscript.

Reference

[1] Fork-and-Flush: Escaping Idea Basins in Autoresearch Agents. arXiv preprint arXiv:2610.07447.

[2] Scaling Discovery Requires Forgetting. arXiv preprint arXiv:2610.20563.

[3] Reinforcement Learning via Self-Distillation. ICML, 2026

[4] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024

[5] On-Policy Distillation. Thinking Machines blog.

W3: The claim of comparable training time is not backed by measurements that account for the additional teacher computation.

We thank the reviewer for raising this question. Our experiments use 20 NVIDIA GH200 GPUs for BrowseComp-plus and 32 for SWE-agent. The total GPU hour spent for training is reported below:

BrowseComp-plusSWE-bench Verified
GRPO599.7748.5
RECAP519.8 (−13.3%)629.6 (-15.8%)

Table 4: Total training time (h) for GRPO and RECAP.

We find that RECAP requires 13.3% and 15.8% lower training time on BrowserComp-plus and SWE-agent, respectively. These results are consistent with the improved interaction efficiency shown in Figure 2, where the RECAP trained model uses fewer tool calls and tokens per trajectory. We’ll include these results in the revised manuscript.


W3: The experiments are limited to a single 4B backbone.

In response to the reviewer’s suggestion, we’ve conducted an additional experiment of Qwen3.5-9B on BrowseComp-plus. The results are summarized below:

Qwen3.5-4BQwen3.5-9B
Base40.747.0
+GRPO58.257.7
+RECAP62.865.7

Table 5: Experiments on other backbone.

The results show that on Qwen3.5-9B RECAP outperforms GRPO by 8.0%, demonstrating that its effectiveness extends beyond the original 4B backbone.


Q1: Can you provide a case study comparing RECAP and GRPO summaries and their effects on subsequent actions?

We thank the reviewer for the suggestion. Below, we provide a case study from SWE-bench Verified.

The task was to fix whitespace in Django’s index SQL. Before their first compaction, both GRPO and RECAP correctly identified the expected order as text_pattern_ops DESC.

GRPO:

  • Summary: Includes an incorrect example with the reversed order: "DESC text_pattern_ops".
  • Subsequent action: Adds a test assertion requiring the same reversed order:
    self.assertIn('DESC text_pattern_ops', sql_desc)
    
    The edit succeeded; the test was subsequently skipped.
  • Task outcome: Failed.

RECAP:

  • Summary: Preserves the correct order text_pattern_ops DESC.
  • Subsequent action: Checks the correct order:
    Result: ' text_pattern_ops DESC'
    Expected: 'text_pattern_ops DESC'
    Pass: True
    
  • Task outcome: Succeeded.

In this example, GRPO’s summary contains incorrect information that is reflected in its subsequent actions, whereas RECAP preserves accurate information and uses it correctly. We’ll include the case study in the revised manuscript.

Comments on the paper

1 comment

Dear Reviewers,

We thank the reviewers for their valuable feedback. We have revised the manuscript to include additional experiments, clarify implementation details, and add case studies suggested by the reviewers. We briefly summarize our updates below:

  • Is full-context policy a reliable teacher? We compared the performance of full-context policy across several configurations and found that the fully-context policy generally achieved higher task success and could serve as a reliable reference to the compacted policy. (Appendix D.1)

  • What happens when the teacher is weaker? We conducted additional experiments by continuing training from a BrowseComp-Plus checkpoint with a frozen teacher. We found that the summary reward alone improved task success, whereas post-compaction action distillation alone reduced it. We further clarified the role of behavior distortion as an auxiliary objective alongside task reward. (Section 4.1 and Appendix D.2)

  • Comparison to additional models and baselines. We added comparisons with sliding-window truncation and context folding. We included additional results with Qwen3.5-9B and CompactionRL. We show that RECAP generalizes to a larger model size and is compatible with other task-reward-only training methods. (Appendices D.3–D.5)

  • Implementation and training cost. We clarified the stop-gradient operation for the per-token estimator and length normalization in distortion estimation. We reported the training cost and showed that RECAP used 13.3% fewer GPU-hours on BrowseComp-Plus and 15.8% fewer on SWE-bench Verified compared to the GRPO baseline. (Appendices B and C)

  • Case study: We added examples in SWE-bench Verified and BrowseComp-Plus to illustrate how RECAP and GRPO behave differently in the summaries and their influence to subsequent actions. (Appendix F)

As the discussion period is approaching its end, we'd really appreciate any feedback from the reviewers on our response and we're happy to address any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.