- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the training of autonomous context folding in agentic RL, highlighting that policy-generated summaries become part of the state that shapes subsequent decisions. To address the resulting credit dilution, replay mismatch, and training costs, FoldAct combines separate summary/action PPO losses, consistency self-distillation against the full context, and selective segment training that randomly samples turns for replay. Experiments show that this approach substantially improves training stability and provides some speedup while maintaining competitive task performance.
Critical strengths
-
The paper is clearly presented, with its three methodological components directly addressing the three challenges identified in the motivation. This alignment makes the design rationale and each component’s role easy to understand.
-
FoldAct reduces update time and memory usage while maintaining competitive task performance. These gains make training context-folding agents more practical under limited computational resources.
Critical weaknesses
-
Coarse summary reward. The summary reward provides no positive retention credit for summaries that omit the final answer but preserve evidence essential to later reasoning. This may underreward useful memory and favor answer-string retention over broader informational value.
-
Mismatch between the consistency loss and the stated problem. The consistency loss addresses differences between folded and full contexts, whereas the motivation attributes replay mismatch to differences between the updated and rollout policies. The paper does not establish whether reducing the former discrepancy actually mitigates the latter.
-
Uniform turn sampling is not justified. Uniform random turn sampling ignores differences in learning value across turns. Since targeted sampling with appropriate importance weighting can also preserve unbiasedness, the paper does not establish whether uniform sampling offers advantages beyond simplicity.
Questions for authors
-
Please compare unified and separate loss normalization under a shared outcome reward, alongside the summary reward shaping variants. This would isolate the contributions of loss normalization and reward shaping to the reported gains.
-
Please provide a derivation or direct empirical evidence that the consistency loss reduces replay mismatch between the updated and rollout policies. This would clarify whether cross-context alignment addresses the policy-distribution mismatch identified in the motivation.
-
I would appreciate comparisons with alternative turn selection strategies under matched replay budgets. These comparisons would clarify whether uniform random selection offers a favorable trade-off between task performance, training stability, and computational cost, including selection overhead.
-
Please consider an adaptive consistency loss that filters or downweights unreliable supervision from the raw context. This could reduce interference from irrelevant information, erroneous reasoning, and tool noise that may persist even in fuller contexts.