acceptodds
Under review as a conference paper at ICLR 2027

Learning Residual Memory of Contextual Compaction

Abstract

Long-horizon agents compact their history to fit a limited context window. Text summaries can lose details and links between observations and decisions, leading to repeated work and errors. We introduce REMORY, a neural memory network that encodes the residual memory of contextual compaction as soft tokens. At each compaction, it reads the history and summary to construct memory for the frozen actor. We train REMORY with a shared two-stage curriculum: text reconstruction followed by summary-conditioned continuation. On SummHay, adding REMORY to the same summary improves citation F1 by +4.04 points and joint score by +3.55, using 5.2% of the full-context positions. With Qwen3.8-27B, REMORY improves scores by 3.0–9.8 points over the baseline across four agent benchmarks. With GLM-5.3-Flash, REMORY improves scores by +6.0 points on BrowseComp and +3.3 on Terminal-Bench 2.1, reaching scores comparable to published frontier LLM reference scores under their respective evaluation protocols.

Primary area:
foundation or frontier models, including LLMs
Keywords:
context compaction, long horizon agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9UAeOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks to use soft token compression to fill in the gaps left behind by hard token compression (compaction). It does this by using DFlash models to produce soft tokens which are put alongside the hard tokens from compaction. The authors use reconstruction and summary conditioned reconstruction training data to do student teacher logit distillation to train the DFlash models. The authors consider Qwen-3.8-27b and GLM-5.3 Flash in their experiments. The authors evaluate on SummHay, comparing to regular compaction (summary) and summary++ which is token matched to the Remory method. Exactly how token matching is enforced is unclear. On SummHay the citation F1 is increased when using Remory. The authors then do a more substantial evaluation over BrowseComp, Terminal-Bench, AutomationBench and JobBench using the codex harness. Remory shows large gains on AutomationBench and JobBench, gains on the other benchmarks are smaller. The authors present LiveTrading and sam-cell-seg as examples of how Remory works, showing less compactions during LiveTrading

Critical strengths
  1. The paper studies using hard and soft token compaction in tandem, a possibly very impactful direction
  2. There are a wide range of benchmarks used for evaluation.
  3. The work suggests only a small number of additional parameters are required to handle soft token compaction.
Critical weaknesses
  1. The introduction is very over anthropomorphised for the experiments that follow.
  2. There is a lot of detail missing from the main text, which even the appendix doesn’t resolve.
  3. It is unclear if any of the baselines are also trained, I believe they are not. If only the Remory method is trained in the harness (or using prompts similar to those used for compaction in the harness) this is quite a large advantage. In any case, models post trained on the same data as Remory for the same number of steps is crucial to ensuring the gains shown in this paper are from Remory and not from additional training. The exact harness and training details need to be a lot more obvious.
  4. The overall time and memory costs of Remory aren’t discussed.
  5. The exact design decisions around the architecture is unclear. For example, how the 1:16 compression ratio was chosen, or why reverse KL was chosen.
  6. The exact implementation of the architecture is unclear. For example, how the 1024 hard tokens -> 64 soft tokens is actually implemented using the DFlash models. Additionally, if the input is 4096 tokens is this 256 soft tokens output int he sequence dimensions or [4,64] tokens putting each chunk of 1024 unto the batch dimension.
  7. An obvious baseline of soft token compression only is missing.



Overall, I think this works most important problem is the lack of compute matched baselines and the lack of detail in the paper to determine how the method is motivated and actually works. I think the core idea of combining hard and soft compression is a good one, just a lot more work needed to prove the idea is as good as what the paper currently claims.

Questions for authors
  1. Are the DFlash models initialised from scratch or fine-tuned?
  2. In Section 4.1 it is detailed Qwen3.8-27B is inferences in extra high mode but with temperature 0. The official Hugging Face page recommends temperature 1.0 when in thinking mode. Why do you chose temperature 0?
  3. If you’re only training the DFlash models, how do the main models learn to use the soft tokens. Could this be improved by also training the main models at the same time?
  4. Why are different decoding strategies used across benchmarks?
  5. Why are standard errors reported in table 2 but not table 3?
3 comments

I maintain my score of rejection for several reasons:

  1. The code and/or enough detail for the method was not made available during review meaning details could not be taken into consideration during the review process.
  2. No other baselines are trained meaning the improvements of the method cannot be distinguished for training the harness used to be in distribution. My belief this may be the case is reinforced by the actor model not receiving any updates during training.
  3. No new ablations were added to defend any of the initial weaknesses highlighted.
AuthorsOpenReview ↗

Response to Reviewer 9UAe

The code and/or enough detail for the method was not made available during review meaning details could not be taken into consideration during the review process.

The code is available in the supplementary material. We invite you to inspect the training and compaction implementations directly, including how REMORY constructs memory tokens and passes them to the actor.

No other baselines are trained meaning the improvements of the method cannot be distinguished for training the harness used to be in distribution. My belief this may be the case is reinforced by the actor model not receiving any updates during training.

In your initial review, you wrote, “If only the REMORY method is trained in the harness (or using prompts similar to those used for compaction in the harness) this is quite a large advantage.” Both conditions use the same actor, harness, and compaction prompts. Only the memory encoder is trained. Gradients pass through the frozen actor (see Figure 4) so that REMORY learns to produce memory tokens the actor can already read. Our comparisons show that adding these tokens improves task performance without updating the actor.

We freeze the actor so that REMORY learns to produce memory tokens the existing actor can use. The fact that the actor receives no updates does not support the inference that the gains come from familiarity with the harness prompts.

No new ablations were added to defend any of the initial weaknesses highlighted.

The requested training ablations require separate large-scale training runs and downstream evaluations, which are difficult to complete within the rebuttal period. We stand by the evidence presented in the paper. Across four agent benchmarks, adding residual memory improves performance under the same evaluation conditions. We believe these experiments establish the method’s effectiveness in the settings tested and demonstrate its potential for long-horizon agents. An exhaustive evaluation of every design choice is a broader research undertaking.

AuthorsOpenReview ↗

To Area Chair, Senior Area Chair, and Program Chairs

We are concerned that Reviewer @9UAe’s assessment rests on a misunderstanding of how REMORY works. The reviewer cites the frozen actor as reinforcing the concern that the gains come from familiarity with the harness prompts. As we explained in our response, gradients pass through the frozen actor to train the memory encoder, which learns to produce tokens the actor can use.

Prior work has shown that frozen language models can use continuous embeddings that do not correspond to any single vocabulary token, including learned soft prompts, embeddings from a separately trained vision encoder, and the probability-weighted token mixtures used in Soft Thinking$^{[1][2][3]}$. In particular, Tsimpoukelli et al. trained an encoder by backpropagating through a frozen language model. Freezing the actor therefore gives no reason to attribute the gains to familiarity with the harness prompts.

We find it frustrating that this misunderstanding persists after clarification, particularly given the reviewer’s Overall Assessment of 1 and Confidence rating of 4. We respectfully ask the committee to consider this when weighing the review.

[1] Lester, Al-Rfou, and Constant (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP.

[2] Tsimpoukelli et al. (2021). Multimodal Few-Shot Learning with Frozen Language Models. NeurIPS.

[3] Zhang et al. (2025). Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space. NeurIPS.

Best,
The Authors

Reviewer_aP2COpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Remory trains a small memory network that encodes what an LLM agent's compaction summary loses, as soft tokens read alongside the summary. It is distilled from a frozen actor's full-history predictions. Validation: SummHay citation gains, plus agent benchmarks, an action probe, and Live Trading.

Critical strengths

This paper's training objective corresponds to the problem it aims to solve, and the cost of integrating it is low. Paper defines information lost to compaction as the residual, namely the predictive information that the summary fails to preserve. It uses the frozen actor's full-history predictions as the teacher and trains a student, which receives the summary and the memory, to approximate them, with a reverse KL loss over the full vocabulary. The actor's weights and tool interface remain unchanged, and the memory network has only 1.94B (Qwen) and 1.502B (GLM) parameters, roughly 7% of the actor, and it runs alongside the existing /compact summary without requiring any replacement of the current workflow. Compared with methods such as Gist Tokens and ICAE, this design keeps a readable text checkpoint and adds a learned supplementary channel on top of it, which gives it a distinct position.

Critical weaknesses

My first concern is that the claimed gap from adding memory lacks more solid support. Paper's main line is that a learned residual memory helps a frozen actor continue working after compaction, but Terminal-Bench 2.1 and BrowseComp each have only a single run, while the standard deviations over five runs on AutomationBench and JobBench lie between 1.0 and 1.4, which shows that run-to-run fluctuation alone exceeds one point. Long-trajectory agent tasks are highly sensitive to sampling and the environment, so a gap of 3 to 6 points from a single run cannot be treated as a stable improvement. When I look back to Section 4.1 and Table 1, the only matched-budget text control (summary++) appears only in SummHay, the improvement there is mainly in citation F1 (+4.04), with coverage almost unchanged (+0.40), while the joint-score interval is computed over 10 document collections and its lower bound is only +0.74. Benchmarks contain no equal-budget text summary, longer summary, or retrieval control, and compare only with and without memory. The memory-augmented input averages 5,024 positions, which is 3.6 times the summary without memory (1,393), so I feel it is impossible to tell whether the gain comes from the learned residual or from simply providing more context capacity. Paper introduces a two-stage curriculum, hierarchical merging, and cross-attention to the summary, among other designs, but apart from an initialization comparison on training agreement, it offers no ablation on downstream performance. The Live Trading ROI gap (−11.86% versus +5.11%) comes from one account per condition, and market choices and settlements evolve independently in each, so it reads more like a description of a phenomenon than evidence.

My second concern is that the paper claims Remory lowers cost and approaches frontier performance, but doesn't count the overhead of constructing the memory. The generation cost reported in the paper includes only the tokens of actor and summary calls, it excludes Remory's execution, source-feature encoding, and tools. However, according to Appendix A, constructing the memory requires the frozen actor to first encode the complete history, feed features from five actor layers into a 1.94B-parameter memory network, and, during hierarchical merging, pass the soft tokens through the actor once more. Compaction therefore has to reprocess the entire history. The conclusion of using only 5.2% of the full-context positions counts only the final input. Conclusions such as lower cost and Remory predicting in parallel with 7% of the parameters therefore can't show the whole system is cheaper. Abstract's claim of scores comparable to frontier models compares against numbers published by each vendor under its own evaluation environment and protocol, whereas Remory's results were obtained in the authors' unified Codex CLI environment, paper itself acknowledges these are only external references. In addition, the memory network must read the target model's hidden states and be trained separately for each actor, so it does not apply to models accessible only through an API, and the limitations section mentions this in just one sentence.

Questions for authors
  1. Remory's input averages 5,024 positions versus 1,393 for the plain summary. In the agent benchmarks, how does it compare against an equal-budget text control (a longer summary or retrieval), and what do the Stage I, Stage II, and hierarchical-merging components each contribute downstream?

  2. Generation cost excludes Remory's execution and source-feature encoding, but building memory requires actor to encode full history. What is the total compute or latency per compaction, including memory construction?

1 comment
AuthorsOpenReview ↗

More Experiments

We evaluate GLM-5.3-Flash on DeepSWE$^{[1]}$ and Agents’ Last Exam (ALE)[2]^{[2]}. Individual tasks rarely fill the model’s context window (1M), so we ran all tasks in each benchmark within a single persistent session. The main agent used spawn_agent to dispatch tasks asynchronously. As each subagent finished, the main agent used /compact to incorporate that subagent’s memory into the shared session. This setup simulates long-running user sessions in which context from earlier tasks accumulates and can introduce noise into later tasks.

BenchmarkWith residual tokensWithout residual tokens
DeepSWE59.352.2
ALE26.520.6

These results suggest that REMORY helps the agent handle cross-task noise in long-running sessions.

[1] Huang, Lee, Tng, and Ge (2026). DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks.

[2] Sun et al. (2026). Agents’ Last Exam.

Reviewer_EC2oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes adding a learned latent memory (REMORY) alongside the text summary the agent produces when it compresses long histories. Remory is a neural network trained so that, when conditioned on its soft token output and the text summary, it matches the agent's next-token predictions given the full history. The paper shows strong gains on agent benchmarks including BrowseComp and Terminal-Bench 2.1.

Critical strengths

The training objective targets exactly the information left behind by compaction, with no hand-chosen notion of what is worth keeping. It also works with existing harnesses without modification, making it a practical add-on for any open-weight agent that compacts its context. This, together with the strong experimental results, makes REMORY a promising direction for deployed agents.

Critical weaknesses
  • BrowseComp and Terminal-Bench 2.1 are run once per configuration. These benchmarks are quite noisy, and a single run usually cannot distinguish the reported gains from run-to-run variation. Repeated runs or a paired per-task test would address this concern.
  • The paper does not compare to previous methods for producing soft tokens (e.g., AutoCompressors, Gist Tokens, ICAE) so it is unclear whether this residual objective compares to those methods.
Questions for authors
  • How are the soft tokens passed to the model at inference time inside Codex CLI? Did you implement a custom serving layer?
  • What is the compute / time cost of building the memory?
1 comment
AuthorsOpenReview ↗

More Experiments

We evaluate GLM-5.3-Flash on DeepSWE$^{[3]}$ and Agents’ Last Exam (ALE)[4]^{[4]}. Individual tasks rarely fill the model’s context window (1M), so we ran all tasks in each benchmark within a single persistent session. The main agent used spawn_agent to dispatch tasks asynchronously. As each subagent finished, the main agent used /compact to incorporate that subagent’s memory into the shared session. This setup simulates long-running user sessions in which context from earlier tasks accumulates and can introduce noise into later tasks.

BenchmarkWith residual tokensWithout residual tokens
DeepSWE59.352.2
ALE26.520.6

These results suggest that REMORY helps the agent handle cross-task noise in long-running sessions.

[3] Huang, Lee, Tng, and Ge (2026). DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks.

[4] Sun et al. (2026). Agents’ Last Exam.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their comments.

  • Reviewer EC2o called REMORY “a promising direction for deployed agents.”
  • Reviewer aP2C highlighted how closely the training objective matches the problem and how the agent can use learned memory alongside a readable summary.
  • Reviewer 9UAe described combining hard and soft tokens as “a possibly very impactful direction” and noted the range of benchmarks we evaluated.

REMORY improves agent performance in very long sessions and can reduce repeated work and generation costs. Qwen3.8-27B + REMORY-1.9B also scored 29.33 in the official ASI-Bench evaluation, the highest score among models of comparable size. We are actively pursuing broad collaborations with industry to develop and deploy it further. During the discussion, we responded to each concern and provided additional experimental evidence. We were disappointed to see one reviewer lower their score without explaining what prompted the change.

ICLR has long welcomed new research directions alongside empirical and practical contributions. To our knowledge, REMORY is the first attempt to learn the residual left by /compact, conditioned on the summary the agent keeps. Intuition guided several of our design choices, including the 1:16 compression ratio. To make it easy for others to try REMORY and build on it, we provide an efficient inference engine and a single command, /remory train, that enables test-time training within a Codex CLI session.

We believe this direction will interest researchers working on long-horizon agents. We would be grateful if the Area Chair, Senior Area Chair, and Program Chairs would consider the paper, our replies, and the additional experimental evidence together when making their assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.