acceptodds
Under review as a conference paper at ICLR 2027

Training with Recoverable Context Improves Agents Beyond Recall at Inference

Abstract

Context management shapes not only what an agent can access at inference time but also what it can learn during reinforcement learning. When truncation causes all rollouts in a group to receive the same reward, group-relative advantage estimation removes that group’s learning signal. We ask whether making truncated observations recoverable during training can improve the learned policy beyond the immediate benefit of recall at inference. Our method stores oversized observations outside the active context, replaces them with compact references, and allows the policy to retrieve them under explicit budgets, while treating recalled text strictly as observation rather than a prediction target. Across four model scales, recall-enabled training improves mathematical accuracy even when recall is disabled at evaluation, indicating that recoverable context changes what the policy learns, not just what it can access. On long-context tasks, it also substantially increases answer coverage. Controlled experiments rule out unprompted recall behavior and generic long-context reasoning as sufficient explanations. These findings establish context management as a training intervention, not merely an inference-time design choice, and highlight the importance of separating its learning effects from its retrieval benefits.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, context management, long-context agents, retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gpqtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper adds recoverable tool observations to Agent0 and studies whether enabling recall during RL improves performance when recall is disabled at evaluation. It reports math gains on Qwen models and increased answer coverage on LongBench-v2.

Critical strengths

Evaluating trained checkpoints with recall disabled is a useful design choice. The Qwen results provide evidence that the intervention affects the learned policy beyond immediate access to retrieved text.

Critical weaknesses
  1. The experiment does not establish why recoverable context is needed. Roughly 14% of training examples are deliberately constructed to be impossible for the truncation baseline (§5.2, Appendix B). The comparison therefore gives one method access to useful training examples that the other cannot solve by construction. There is no larger-observation-budget control or experiment removing this synthetic subset; §9 explicitly acknowledges the former omission. The results cannot distinguish a benefit specific to recoverable context from the simpler benefit of making more training examples solvable.

  2. The proposed learning-signal mechanism is not measured. The explanation rests on converting zero-advantage groups into useful training signal, but Table 1 reports task success, KL divergence, and response length rather than the frequency of groups with nonzero reward variance. Higher success does not establish this mechanism: uniformly successful groups also have zero advantages. Moreover, §6.3 treats final memory-stress performance as the amount of recovered signal, although the policy reflects updates accumulated throughout training. These proxies do not substantiate the paper’s central explanation for the downstream gains.

Questions for authors
  1. Does the math improvement with recall disabled persist when the synthetic memory-stress subset is removed, or when the baseline can directly observe the same evidence?

  2. Do training logs show that recall increases the proportion of groups with nonzero reward variance, separately for synthetic and ordinary math tasks?

1 comment

Thank you for these questions.

Q: The experiment does not establish why recoverable context is needed.

A: In our log files, 200 of 1,400 training examples (14.29%) are memory-stress examples. These examples are structurally unsolvable under the 512-token destructive-truncation condition but potentially solvable with recall. Our evaluation with recall disabled establishes that the effect is learned during training rather than being solely an inference-time retrieval benefit. It does not, however, establish that recoverability itself is responsible for the improvement, because the synthetic subset remains present during training.

Q: The proposed learning-signal mechanism is not measured.

A: we agree. Table 1 establishes differences in training outcomes, but it does not directly measure the proposed learning-signal mechanism. In particular, higher task success does not imply nonzero within-group advantages because both uniformly unsuccessful and uniformly successful groups have zero reward variance.

Q: Does the math improvement with recall disabled persist when the synthetic memory-stress subset is removed, or when the baseline can directly observe the same evidence?

A: Our current experiments do not include the controls or logging needed to answer them conclusively. First, the reported math improvement with recall disabled is measured after training on the complete mixture: 600 GSM8K examples, 600 MATH examples, and 200 synthetic memory-stress examples. Thus, the synthetic subset constitutes 14.3% of training. We did not train a matched pair with this subset removed, nor did we train a truncation baseline with a sufficiently large observation budget to expose the same evidence. Consequently, the current results do not establish whether the residual math improvement is specific to recoverable context or simply results from making more training examples solvable.

Q: Do training logs show that recall increases the proportion of groups with nonzero reward variance, separately for synthetic and ordinary math tasks?

A: The current logs do not directly record the proportion of rollout groups with nonzero reward variance. They contain aggregate reward, validation performance, self-consistency, and recall-use statistics, but not the complete per-group reward vector joined with task type.

Reviewer_zW4KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the important problems of truncation of rollouts during RL training. Standard GRPO effectively ignores data points whose rollouts are all truncated. To enable learning on such data points the authors propose a system that places truncation candidate rollouts to an external store and gives the policy to recall them as suitable. The authors then show that this is effective in improving the performance especially on long context tasks at inference time at various model/policy sizes.

Critical strengths
  • The authors motivate the problem well - specifically the issue around the length of tool call response and how truncation can lead to homogeneously incorrect answers form the policy and thus zero reward variance and learning signal.
  • The main design choices of compressing overflowing tool responses instead of throwing them away and keeping this compression process out of main task learning seems different from prior works.
Critical weaknesses
  • The main issue is clarity in describing the proposed approach. See several things that are not clear (see Questions section ) but are essential for assess the effectiveness of the proposed approach.
  • Also the empirical results are limited to one dataset, more analysis of other agentic settings will be useful and additional comparison with works that learn to compact during training as well.
Questions for authors

How does variations in how agent0’s compaction affect the overall system?

Line 173 and 174 are not clear.

How does bounded recall ensure the same information/ learning signal loss as the truncation of tool responses? Is this possible because we do iterative recall of the whole response in chunks and remove irrelevant chunks? How does this removal affect kv-cache etc and how is decided what chunk gets removed?

In line 251-252 – what does protocol; call not disclosed and recall-silent mean?

1 comment

Thank you for identifying these ambiguities.

Q: How do variations in Agent0’s compaction affect the system?

A: Agent0 does not perform semantic “compaction” in our implementation. Its primary loss mechanism is per-observation tail retention: if a tool response exceeds length of observation, only its final tokens are retained. A separate model-context limit bounds the complete trajectory. Different truncation and recovery configurations produce different information–computation trade-offs. Increasing the observation budget exposes more evidence directly but increases prompt length, KV-cache use, and rollout cost. Keeping the prefix, tail, or both favors evidence occurring at different positions. In our method, larger previews or recall-return budgets expose more evidence but consume more context and reduce the remaining generation budget.

Q: Clarifying Lines 173–174

A: Unlike approaches that train a separate context manager while freezing the task agent, we let the executor itself choose whether to recall an observation. The RL loss is applied to executor-generated tokens, including the recall action, but not to citations, tool outputs, or recalled text, because these are generated by the environment. Recalled text can still influence gradients indirectly by conditioning later policy actions. This masking rule prevents environment text from being treated as a policy prediction target; we do not claim that it eliminates gradient interference or improves performance independently, because we do not ablate that mechanism.

Q: What bounded recall actually does?

A: Bounded recall is not intended to reproduce the same information loss as truncation. Its purpose is to retain the original observation externally while limiting how much of it is placed in the model’s active context at once.

Q: What “protocol not disclosed” and “recall-silent” mean?

A: “Protocol not disclosed” means that the recovery mechanism remains enabled. Oversized observations are still archived and replaced by an ID-bearing citation, and the environment still recognizes "recall" action. However, the system prompt, task prompt, and citation contain no recall instructions or action syntax. We call this inference condition "recall-silent". It tests whether a policy trained with recall continues to issue the learned recall action without an evaluation-time reminder. By contrast, in the recall condition, the same mechanism is enabled and its syntax is explicitly disclosed.

Reviewer_T3x8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether making truncated extensively long tool responses recoverable during training can improve the learned policy. It proposes recoverable context, allowing the model to retrieve an archived observation through an explicit recall action specifying its identifier.

Critical strengths

The method is simple, straightforward and easy to follow. And the experiments show improvements.

Critical weaknesses
  1. The paper does not provide enough background discussion, e,g., under which existing agent harnesses are tool outputs irreversibly truncated? Which harnesses already support revisiting or searching previous contexts (including tool outputs)? Also, there are many different ways to deal with truncation beyond recoverable context, e.g. use an additional model do summarization for overly long tool outputs. Ido not mean your method is not necessary but I think you should at least compare it with those. Also, the scope is narrow. Information loss can involve documents and the accumulated context itself, not only individual tool outputs. I believe the recoverable context should be more useful in those scenarios.

  2. Describing the recall protocol in the prompt is a relatively weak intervention by itself. I expect the paper should teach the model learn to use recoverable context with RL?

  3. The writing has a large room to improve. Under fig1 there is sufficient empty space but the caption is too simple. At least you should tell me where to focus in the figure? The experimental section shows too much redundant information that should be put into the appendix.

  4. The limitation section mentions weaknesses rather than limitations. They should be addressed before submission.

Overall, I think this paper is not ready for submission.

Questions for authors

see weakness

1 comment

Thank you for this careful feedback. We will revise both the empirical comparison and the framing.

Q: The paper does not provide enough background discussion, e,g., under which existing agent harnesses are tool outputs irreversibly truncated? Which harnesses already support revisiting or searching previous contexts (including tool outputs)?

A: We use “irreversible compaction” to mean that the policy cannot recover the exact discarded text during the trajectory, even if it remains in an external debug log. Agent0 applies fixed per-observation truncation without a recovery action. Other harnesses provide strong context management: OpenHands retains an event history while presenting summarized context, and frameworks such as AutoGen and LangGraph support application-defined persistent or searchable memory. These differ from our exact recovery because tool outputs must be explicitly stored and retrieved by the application.

Q: Also, there are many different ways to deal with truncation beyond recoverable context, e.g. use an additional model do summarization for overly long tool outputs.

A: We have run on Qwen3-8/32B (more comparisons on other models and settings will be added in the revised version). We compared inference results on the needle-in-a-haystack task over three seeds (42, 82, and 122), with 1,000 tasks per seed (3,000 attempts per method). No-Ans. denotes the mean number of unanswered tasks; Pass denotes the mean number of successful tasks. Methods: Full Context (no truncation), Sliding Window, LLM Summary, Structured State, RAG Memory, and Ours.

Our method achieves 99.00% accuracy with Qwen3-8B and 99.80% with Qwen3-32B, substantially outperforming the strongest baseline (RAG memory), which achieves 79.57% and 96.67%, respectively.

\hline \textbf{Method} & \textbf{Model} & \textbf{Pass/Att.} & \textbf{No-Ans.} & \textbf{Pass} & \textbf{Acc.} & \textbf{SD} & \textbf{Mean }\pm\textbf{ SD} \\\\ \hline \text{Full_context} & \text{Qwen3-8B} & 1104/3000 & 631.67 & 368.00 & 36.80\\% & 2.59\\% & 36.80\\%\pm2.59\\% \\\\ \text{Sliding_window} & & 1686/3000 & 437.33 & 562.00 & 56.20\\% & 0.90\\% & 56.20\\%\pm0.90\\% \\\\ \text{LLM_summary} & & 2153/3000 & 279.33 & 717.67 & 71.77\\% & 1.36\\% & 71.77\\%\pm1.36\\% \\\\ \text{Structured_state} & & 2045/3000 & 314.33 & 681.67 & 68.17\\% & 1.64\\% & 68.17\\%\pm1.64\\% \\\\ \text{RAG_memory} & & 2387/3000 & 201.00 & 795.67 & 79.57\\% & 0.80\\% & 79.57\\%\pm0.80\\% \\\\ \mathbf{Ours} & & \mathbf{2970/3000} & \mathbf{9.67} & \mathbf{990.00} & \mathbf{99.00\\%} & 0.17\\% & \mathbf{99.00\\%\pm0.17\\%} \\\\ \hline \text{Full_context} & \text{Qwen3-32B} & 88/3000 & 964.67 & 29.33 & 2.93\\% & 0.76\\% & 2.93\\%\pm0.76\\% \\\\ \text{Sliding_window} & & 2860/3000 & 44.33 & 953.33 & 95.33\\% & 0.64\\% & 95.33\\%\pm0.64\\% \\\\ \text{LLM_summary} & & 2817/3000 & 22.33 & 939.00 & 93.90\\% & 0.52\\% & 93.90\\%\pm0.52\\% \\\\ \text{Structured_state} & & 2899/3000 & 15.33 & 966.33 & 96.63\\% & 0.78\\% & 96.63\\%\pm0.78\\% \\\\ \text{RAG_memory} & & 2900/3000 & 12.33 & 966.67 & 96.67\\% & 1.15\\% & 96.67\\%\pm1.15\\% \\\\ \mathbf{Ours} & & \mathbf{2994/3000} & \mathbf{0.67} & \mathbf{998.00} & \mathbf{99.80\\%} & 0.17\\% & \mathbf{99.80\\%\pm0.17\\%} \\\\ \hline \end{array}$$ Q: Is recall only prompted, or learned through RL? A: In our method, the prompt specifies the action interface—analogous to describing any tool—but the executor is then trained with RL using only terminal task reward. Recall actions are policy-generated and receive gradients, whereas retrieved content is treated as an environment observation. Q: Figure and experimental presentation. A: Thank you very much for the comment on our Figure caption. It would be more detailed as: Left: conventional Agent0 retains only the final observations of an oversized tool response and permanently discards the remainder. Right: our recoverable context mechanism stores the complete response in a trajectory-local archive, inserts a citation, and allows the policy to re-materialize the original text through an explicit, budgeted recall action. Both conditions use the same model, optimizer, context ceiling, and terminal outcome reward. Policy-generated reasoning and recall actions contribute to the RL loss; citations, tool observations, and recalled text condition later actions. We will also revise our submission to be more compact, especially in the experimental section. Q: The limitation section mentions weaknesses rather than limitations. A: We agree. The absence of a larger-observation-budget control is a threat to validity, not an intrinsic limitation. We will revise it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.