Learning Residual Memory of Contextual Compaction

Long-horizon agents compact their history to fit a limited context window. Text summaries can lose details and links between observations and decisions, leading to repeated work and errors. We introduce REMORY, a neural memory network that encodes the residual memory of contextual compaction as soft tokens. At each compaction, it reads the history and summary to construct memory for the frozen actor. We train REMORY with a shared two-stage curriculum: text reconstruction followed by summary-conditioned continuation. On SummHay, adding REMORY to the same summary improves citation F1 by +4.04 points and joint score by +3.55, using 5.2% of the full-context positions. With Qwen3.8-27B, REMORY improves scores by 3.0–9.8 points over the baseline across four agent benchmarks. With GLM-5.3-Flash, REMORY improves scores by +6.0 points on BrowseComp and +3.3 on Terminal-Bench 2.1, reaching scores comparable to published frontier LLM reference scores under their respective evaluation protocols.