acceptodds
Under review as a conference paper at ICLR 2027

FoldAct: Stable and Efficient Training for Context-Folding Agents

Abstract

Long-horizon reinforcement learning (RL) for large language model agents operates under fixed context windows while interaction histories keep growing. Context folding summarizes this history, but end-to-end folding changes the training problem because the same policy writes summaries and later acts from them. Summary tokens are policy-written memory—tokens that become future visible state—so a unified token-level objective creates credit dilution, summary-induced replay mismatch, and expensive per-turn replay. We introduce FoldAct, an RL training method for policy-written memory in long-horizon agents. FoldAct combines separated summary/action objectives, a consistency loss that stabilizes behavior after context folding, and selective segment training. At inference, the same policy writes summaries and produces subsequent actions, without a teacher summarizer, external memory model, or change to the deployed architecture. Our controlled claims are feasibility, update-side cost, and stability: FoldAct makes folded training feasible with a 5.19× speedup and stabilizes response length.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, Context Folding Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Zy2dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the training of autonomous context folding in agentic RL, highlighting that policy-generated summaries become part of the state that shapes subsequent decisions. To address the resulting credit dilution, replay mismatch, and training costs, FoldAct combines separate summary/action PPO losses, consistency self-distillation against the full context, and selective segment training that randomly samples turns for replay. Experiments show that this approach substantially improves training stability and provides some speedup while maintaining competitive task performance.

Critical strengths
  1. The paper is clearly presented, with its three methodological components directly addressing the three challenges identified in the motivation. This alignment makes the design rationale and each component’s role easy to understand.

  2. FoldAct reduces update time and memory usage while maintaining competitive task performance. These gains make training context-folding agents more practical under limited computational resources.

Critical weaknesses
  1. Coarse summary reward. The summary reward provides no positive retention credit for summaries that omit the final answer but preserve evidence essential to later reasoning. This may underreward useful memory and favor answer-string retention over broader informational value.

  2. Mismatch between the consistency loss and the stated problem. The consistency loss addresses differences between folded and full contexts, whereas the motivation attributes replay mismatch to differences between the updated and rollout policies. The paper does not establish whether reducing the former discrepancy actually mitigates the latter.

  3. Uniform turn sampling is not justified. Uniform random turn sampling ignores differences in learning value across turns. Since targeted sampling with appropriate importance weighting can also preserve unbiasedness, the paper does not establish whether uniform sampling offers advantages beyond simplicity.

Questions for authors
  1. Please compare unified and separate loss normalization under a shared outcome reward, alongside the summary reward shaping variants. This would isolate the contributions of loss normalization and reward shaping to the reported gains.

  2. Please provide a derivation or direct empirical evidence that the consistency loss reduces replay mismatch between the updated and rollout policies. This would clarify whether cross-context alignment addresses the policy-distribution mismatch identified in the motivation.

  3. I would appreciate comparisons with alternative turn selection strategies under matched replay budgets. These comparisons would clarify whether uniform random selection offers a favorable trade-off between task performance, training stability, and computational cost, including selection overhead.

  4. Please consider an adaptive consistency loss that filters or downweights unreliable supervision from the raw context. This could reduce interference from irrelevant information, erroneous reasoning, and tool noise that may persist even in fuller contexts.

Reviewer_PMpQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes FoldAct, an RL framework for long-horizon search agents that treats context summarization as a special action altering future observations, causing gradient dilution, self-conditioning instability, and high training cost. It addresses these with separated summary/action losses, a full-context consistency KL loss, and selective segment training. Experiments on RAG and web search show better performance, stable training, lower cost, and bounded context growth.

Critical strengths

A main strength of FoldAct is its conceptual clarity and tight problem-to-remedy mapping: it sharply diagnoses why context folding breaks standard RL: summarization reshapes future observations, causing gradient dilution, self-conditioning instability, and high cost, and matches each failure to a concrete fix (separated summary/action losses, full-context consistency KL, and selective segment training). Empirically, it stays stable where unaided training collapses, remains competitive within its paradigm (notably beating a 32B baseline on WebWalker with a 7B model), and cuts training time and memory while bounding context growth. Its core strength is formulating an underappreciated instability problem precisely and validating a targeted stabilization strategy.

Critical weaknesses
  1. A major drawback is that FoldAct overstates the novelty of its core insight: context summarization is a special, non-stationary action requiring stabilization, without adequately distinguishing it from closely related work. Similar tensions and solutions appear in Context-Folding/FoldGRPO (process rewards for folding), AgentFold and Memory as Action (folding as an explicit action), ReSum (periodic summarization with restart), CompactionRL and MemSearcher (stabilizing training under compression), and IterSynth and Memex(RL) (architectural alternatives). FoldAct’s real contribution is a sharper formalization of non-stationarity plus a specific loss-decomposition remedy, but the paper does not clearly establish what is new relative to these lines or why its fix is preferable, weakening its claimed novelty and obscuring the contribution boundary.

  2. Narrow experimental setting limits external validity: The entire empirical case rests on a single model scale (Qwen-2.5-7B-Instruct) and one training-data pipeline (200 trajectories from ASearcher, with summaries generated by a separate Qwen3-30B model). There is no evaluation of whether this three-component recipe generalizes across model families, larger scales, or different summary-generation regimes, making the claimed efficiency and stability gains fragile to distribution shift.

  3. Selective segment training introduces a non-uniform, brittle trade-off. Dropping half the turns (p_drop=0.5) does not improve performance uniformly: WebWalker rises slightly (+0.4) while GAIA drops by 1.3 and XBench-DeepSearch by 2.5. The paper frames this as “mainly saves compute,” but that framing obscures the fact that the method can degrade performance on some benchmarks without a clear criterion for when this will occur, turning a core efficiency component into an unpredictable source of variance.

Questions for authors
  1. What is the essential difference from FoldGRPO? FoldGRPO also brings folding into RL training but uses process rewards to shape folding behavior, whereas FoldAct uses separated losses plus a consistency KL. Can you provide a controlled comparison showing that your approach outperforms the process-reward route under the same model and training data? Without that comparison, where exactly is the boundary of the “first formalization of non-stationarity” contribution?

  2. How were the hyperparameters chosen? i.e., summary rewards (+0.2 / −0.2), p_drop = 0.5 These constants look hand-tuned. How sensitive are the training dynamics to them? If they were changed to +0.1 / −0.1 or +0.5 / −0.5, how much would the collapse point move? Also, why choose p_drop = 0.5? Selective segment training slightly improves WebWalker but degrades GAIA and XBench-DeepSearch. Did you try p_drop = 0.3 or 0.7? Is there a task-dependent principle for choosing the drop ratio, or is it purely trial and error?

  3. Is the “specialness” of the summarization action overstated? Any action that changes the observation distribution, for example: opening a new webpage, calling a retrieval tool, also changes future observations. Why is summarization singled out as a “non-ordinary action”? Is it because its impact is larger, or because of its position in the trajectory? A more precise criterion is needed for which actions require special treatment.

Reviewer_L3axOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FoldAct trains an agent that writes summaries into its own working context and subsequently acts using those summaries. The method combines separately normalized PPO objectives for summary and action tokens, lightweight summary rewards, a consistency loss between folded and recent-history contexts, and random subsampling of turns during replay. All rollout turns are retained; subsampling affects only which turns contribute training losses. The deployed agent uses a single Qwen2.5-7B policy, initialized by SFT on summary-annotated trajectories. Experiments report lower update-side replay cost, more stable response lengths, and answer-quality results on local retrieval and web-search tasks. The paper also gives an occupancy-drift bound and a simplified cost model.

Critical strengths
  1. The paper addresses a concrete difficulty in training agents with self-written memory. Summary tokens affect later decisions despite occupying a relatively small part of the generated text. Separating their objective from ordinary action tokens is a reasonable response to this imbalance. The method also clearly distinguishes skipping a replay loss from deleting a rollout turn, preserving the actual context seen by each retained training segment.

  2. The ablations provide evidence that the components serve different purposes. Removing consistency leads to the unstable response-length behavior in Figure 5, while removing loss separation reduces HotpotQA from 38.5/29.5 to 36.2/27.8 F1/EM. The controlled GAIA comparison reaches 45.0 versus 41.5 for unified PPO folding and 43.0 for the reproduced SUPO baseline. These results suggest that the training changes can help beyond the initial summary-format SFT, without requiring another model at inference.

Critical weaknesses
  1. The headline 5.19× speedup is difficult to interpret. Table 1 gives the recent-history baseline a replay time of 4,846.72 seconds while marking its memory result as OOM. The paper does not explain whether that time comes from a completed update, an extrapolation, or a different memory configuration. Relative to the feasible SUPO row, FoldAct's reported update-time speedup is about 1.30×. That is still useful, but the larger headline result needs a clear denominator. The cost-model comparison also uses a folding ratio of 0.25, whereas Table 5 reports 0.65 for Local RAG and 0.25 only for the longest Web Search bin. Identifying the profiled workload and reporting the corresponding context and response lengths would make both comparisons assessable.

  2. The evidence for the consistency mechanism is incomplete. Figure 5 supports an effect on response length, but gives no task-return or repetition-rate trajectory to substantiate the claimed training collapse. The analytical connection is also limited: Lemma 3.1 compares state occupancy across policies, while Equation 7 compares sampled-token log probabilities under two contexts at the same current parameters. The latter does not directly control the between-update transition-kernel drift in the lemma. I do not expect a full guarantee for this practical loss, but the paper should narrow its mechanism claim or measure the relevant drift. This matters because the consistency branch accounts for most of the remaining update cost. Its reference is also a recent-history window, which can omit older evidence that a folded summary has successfully retained.

Questions for authors
  1. How was the OOM baseline's update time obtained, and did its timed updates complete? Please identify the workload, context limit, effective batch, replay configuration, and per-worker peak memory for Table 1.
  2. Does the 0.25 folding ratio in Corollary 4.1 describe that same workload? What are its context and response-length distributions, and how much of the measured cost comes from the linear versus attention terms?
  3. What happens to task return and repetition during the instability shown in Figure 5, and how repeatable is it across training runs? Evidence of summary-policy or folded-state drift would further support the proposed explanation. Please also clarify how the consistency target handles useful older evidence retained only in the summary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.