- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper looks to use soft token compression to fill in the gaps left behind by hard token compression (compaction). It does this by using DFlash models to produce soft tokens which are put alongside the hard tokens from compaction. The authors use reconstruction and summary conditioned reconstruction training data to do student teacher logit distillation to train the DFlash models. The authors consider Qwen-3.8-27b and GLM-5.3 Flash in their experiments. The authors evaluate on SummHay, comparing to regular compaction (summary) and summary++ which is token matched to the Remory method. Exactly how token matching is enforced is unclear. On SummHay the citation F1 is increased when using Remory. The authors then do a more substantial evaluation over BrowseComp, Terminal-Bench, AutomationBench and JobBench using the codex harness. Remory shows large gains on AutomationBench and JobBench, gains on the other benchmarks are smaller. The authors present LiveTrading and sam-cell-seg as examples of how Remory works, showing less compactions during LiveTrading
Critical strengths
- The paper studies using hard and soft token compaction in tandem, a possibly very impactful direction
- There are a wide range of benchmarks used for evaluation.
- The work suggests only a small number of additional parameters are required to handle soft token compaction.
Critical weaknesses
- The introduction is very over anthropomorphised for the experiments that follow.
- There is a lot of detail missing from the main text, which even the appendix doesn’t resolve.
- It is unclear if any of the baselines are also trained, I believe they are not. If only the Remory method is trained in the harness (or using prompts similar to those used for compaction in the harness) this is quite a large advantage. In any case, models post trained on the same data as Remory for the same number of steps is crucial to ensuring the gains shown in this paper are from Remory and not from additional training. The exact harness and training details need to be a lot more obvious.
- The overall time and memory costs of Remory aren’t discussed.
- The exact design decisions around the architecture is unclear. For example, how the 1:16 compression ratio was chosen, or why reverse KL was chosen.
- The exact implementation of the architecture is unclear. For example, how the 1024 hard tokens -> 64 soft tokens is actually implemented using the DFlash models. Additionally, if the input is 4096 tokens is this 256 soft tokens output int he sequence dimensions or [4,64] tokens putting each chunk of 1024 unto the batch dimension.
- An obvious baseline of soft token compression only is missing.
Overall, I think this works most important problem is the lack of compute matched baselines and the lack of detail in the paper to determine how the method is motivated and actually works. I think the core idea of combining hard and soft compression is a good one, just a lot more work needed to prove the idea is as good as what the paper currently claims.
Questions for authors
- Are the DFlash models initialised from scratch or fine-tuned?
- In Section 4.1 it is detailed Qwen3.8-27B is inferences in extra high mode but with temperature 0. The official Hugging Face page recommends temperature 1.0 when in thinking mode. Why do you chose temperature 0?
- If you’re only training the DFlash models, how do the main models learn to use the soft tokens. Could this be improved by also training the main models at the same time?
- Why are different decoding strategies used across benchmarks?
- Why are standard errors reported in table 2 but not table 3?
3 comments
Response to Reviewer 9UAe
The code and/or enough detail for the method was not made available during review meaning details could not be taken into consideration during the review process.
The code is available in the supplementary material. We invite you to inspect the training and compaction implementations directly, including how REMORY constructs memory tokens and passes them to the actor.
No other baselines are trained meaning the improvements of the method cannot be distinguished for training the harness used to be in distribution. My belief this may be the case is reinforced by the actor model not receiving any updates during training.
In your initial review, you wrote, “If only the REMORY method is trained in the harness (or using prompts similar to those used for compaction in the harness) this is quite a large advantage.” Both conditions use the same actor, harness, and compaction prompts. Only the memory encoder is trained. Gradients pass through the frozen actor (see Figure 4) so that REMORY learns to produce memory tokens the actor can already read. Our comparisons show that adding these tokens improves task performance without updating the actor.
We freeze the actor so that REMORY learns to produce memory tokens the existing actor can use. The fact that the actor receives no updates does not support the inference that the gains come from familiarity with the harness prompts.
No new ablations were added to defend any of the initial weaknesses highlighted.
The requested training ablations require separate large-scale training runs and downstream evaluations, which are difficult to complete within the rebuttal period. We stand by the evidence presented in the paper. Across four agent benchmarks, adding residual memory improves performance under the same evaluation conditions. We believe these experiments establish the method’s effectiveness in the settings tested and demonstrate its potential for long-horizon agents. An exhaustive evaluation of every design choice is a broader research undertaking.
To Area Chair, Senior Area Chair, and Program Chairs
We are concerned that Reviewer @9UAe’s assessment rests on a misunderstanding of how REMORY works. The reviewer cites the frozen actor as reinforcing the concern that the gains come from familiarity with the harness prompts. As we explained in our response, gradients pass through the frozen actor to train the memory encoder, which learns to produce tokens the actor can use.
Prior work has shown that frozen language models can use continuous embeddings that do not correspond to any single vocabulary token, including learned soft prompts, embeddings from a separately trained vision encoder, and the probability-weighted token mixtures used in Soft Thinking$^{[1][2][3]}$. In particular, Tsimpoukelli et al. trained an encoder by backpropagating through a frozen language model. Freezing the actor therefore gives no reason to attribute the gains to familiarity with the harness prompts.
We find it frustrating that this misunderstanding persists after clarification, particularly given the reviewer’s Overall Assessment of 1 and Confidence rating of 4. We respectfully ask the committee to consider this when weighing the review.
[1] Lester, Al-Rfou, and Constant (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP.
[2] Tsimpoukelli et al. (2021). Multimodal Few-Shot Learning with Frozen Language Models. NeurIPS.
[3] Zhang et al. (2025). Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space. NeurIPS.
Best,
The Authors
I maintain my score of rejection for several reasons: