- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CompactionRL, a PPO-based reinforcement learning framework for long-horizon LLM agents under a fixed context budget. When the context nears its limit during rollout, the policy generates a summary and resumes from the summary plus recent turns. Both execution and summarization tokens are optimized under the same task reward. The authors address compacted-rollout issues with single-rollout PPO, token-level loss normalization, and cross-segment GAE. Validation on SWE-bench Verified and Terminal-Bench 2.0 shows gains over base models with inference compaction, standard no-compaction RL, and a variant without summary loss. Results suggest that learning compaction during RL improves long-horizon agentic coding performance without increasing peak context length.
Critical strengths
1.Jointly optimizing execution and summarization under a shared task reward is a strong practical contribution. It turns context compaction from an inference-time heuristic into a trainable policy component. Ablations show that summary loss matters, which is consequential for long-horizon agentic RL under fixed context budgets.
2.The optimization methods for compacted rollouts are technically meaningful. Token-level loss normalization reduces segment-count bias, and cross-segment GAE improves credit assignment across compaction boundaries. Ablations support both, strengthening the framework beyond a simple application of PPO.
Critical weaknesses
The empirical evaluation is primarily conducted on coding and terminal-style tasks with models from a single family. While the results are promising, it remains open how well the approach generalizes to other long-horizon agentic settings and to models with different architectures or scales. A broader evaluation would help clarify the scope of the claimed benefits.
Questions for authors
In training, summary tokens and execution tokens share the same final task reward. How does the method prevent the policy from learning to produce longer or more detailed summaries simply to gain credit, rather than because they improve continuation? Is there any length, quality, or efficiency regularization?
2 comments
Reply to author
Thank you to the author for the response; my concerns have been addressed.
This evaluation scope is consistent with prior RL studies: DAPO and VAPO study mathematical reasoning with a single Qwen2.5-32B backbone [1,2], while RLEF, Self-play SWE-RL, Agent-RLVR, and Seed-Coder etc.. primarily evaluate on coding tasks [3–6].
CompactionRL is task-agnostic and requires no GLM-specific architecture, nevertheless, it achieves improvements at both 30B and 106B scales, including on Terminal-Bench 2.0 after training only on SWE-Dev. These results provide evidence of effectiveness across model scales and generalization beyond the training benchmark.
[1] Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476.
[2] Yue et al. VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arXiv:2504.05118.
[3] Gehring et al. RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning. arXiv:2410.02089.
[4] Wei et al. Toward Training Superintelligent Software Agents through Self-Play SWE-RL. arXiv:2512.18552.
[5] Da et al. Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards. arXiv:2506.11425.
[6] ByteDance Seed et al. Seed-Coder: Let the Code Model Curate Data for Itself. arXiv:2506.03524.
The policy has no incentive to generate longer summaries merely to gain credit, as the reward depends solely on task correctness, not length. If leading to failed trajectories, longer summaries would instead receive greater negative penalties. We use no explicit length penalty or auxiliary summary-quality reward. Appendix C in our paper evaluates summary quality using a rubric covering fidelity, failure history, continuation usefulness etc... CompactionRL achieves a higher overall quality score (3.52 vs. 3.30), supporting that the learned summaries are of better quality and better facilitate subsequent task execution.