- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an approach to memory for agents that uses recursive compression to store important information efficiently. It evaluates this architecture on the symbolic alchemy task, and tests it against a range of other (reimplemented) memory architectures.
Critical strengths
- The exact proposed memory mechanism seems novel
- The experimental validation on is broad; I appreciated the evaluation of additional metrics on Alchemy like prediction / recall NLL as well as accuracy, the baselines compared, and the ablations.
Critical weaknesses
I have a number of concerns about the evaluation.
-
ProcGen: I suspect the policy (for both this model and all the baselines) is substantially undertrained. The original paper used up to 200m timesteps. Comparing the policies a few percent of the way into training will yield strongly biased conclusions.
-
Even on alchemy, the results seem suspiciously weak. They are not directly comparable to Wang et al.'s original results (due to changing # of trials in a given chemistry and # of potions) but the return seems far from what could optimally be achieved, and maybe comparable to a random heuristic (one of the baselines Wang et al. considered). It would be easy to score against the alchemy baselines (which were open-sourced) and it would be useful to know if the policy actually improves on the no-meta-learning random heuristic (that just learns values), and if so, by how much.
- likewise baselines like no-memory QA on LongMem are not reported for comparison.
-
Many essential details are missing. For example, I am concerned about the amount of training on ProcGen (as noted above) but I can’t even assess whether I should be concerned about it on the other environments because (as far as I can tell) absolutely nothing is said about how much the models are trained. And many other details like what tier of LongMem / reader / etc. are not stated.
Questions for authors
- can you clarify all the missing details about training, and e.g. how you decided to stop / when training was sufficient?
- Can you compare to the alchemy baselines appropriately, and assess how much your model improves over the simple heuristic baselines?
- Can you rerun experiments on procgen (and other tasks, as appropriate) with a reasonable length of training?
9 comments
Response to Reviewer Yhf7 (2/2)
W3. No-memory baseline on LongMemEval-V2
The frozen reader answers 23 of 451 questions without memory, and RPC raises this to 47, a 2.04-fold increase. As you suggested, we ran the same reader on all 451 questions with an empty memory:
| Method | Correct of 451 | Accuracy | Gain over no memory |
|---|---|---|---|
| No memory | 23 | 5.10% | 0 |
| BM25 | 26 | 5.76% | +3 |
| H-MEM | 33 | 7.32% | +10 |
| Titans | 41 | 9.09% | +18 |
| RPC | 47 | 10.42% | +24 |
The other baselines (DeMem, RGMem, and MemTree) answer 22 to 30 questions and are reported in Figure 8a.
Takeaway: RPC adds 24 correct answers over the reader alone, a paired gain of 5.32 percentage points [95% CI: 2.66, 8.20]. Moreover, without memory the reader returns "unknown" on 63.4% of the questions, so correct answers depend on retained content. RPC's QA count is therefore well above what the reader achieves from its own prior.
In the updated manuscript, Section 4.4 adds the no-memory reference and the paired interval, and Table 6 (Appendix G) compares RPC with every method.
W4 & Q1. What are the training amounts, stopping rules, and model settings?
Memory forms online during evaluation, so there is no training to stop. Every trained component uses a budget fixed in advance, and the table below gives the training and evaluation settings for each benchmark.
The main trained model is the Procgen controller. We train it for 5.0M frames and freeze the checkpoint with the best return on development levels (at 4.0M frames), which are disjoint from the evaluation levels, before any memory method runs. To check whether this budget is enough for comparing memory methods, we trained 16 to 32 times longer, and the conclusions hold, as shown in our response to W1 & Q3.
| Benchmark | What is trained, and how much | Stopping rule | Evaluation settings |
|---|---|---|---|
| Alchemy | Shared outcome prior, fitted once on 128 training worlds (5,120 transitions) | Fixed data budget | Official symbolic environment; 5 trials, 3 stones, 8 potions, at most 20 actions per trial; 10 paper worlds and 128 additional worlds; 64 KiB cap |
| Controlled suite | No shared model; memory forms online from 128 formation and 96 evidence observations per condition | Fixed data budget | 128 test observations per condition; 25 conditions, 10 seeds; caps of 8, 16, and 32 KiB |
| LongMemEval-V2 | No shared model; the reader, encoder, and judge are frozen | No training | Medium tier, all 451 questions; reader Qwen3.5-9B, the benchmark's official reader (4-bit, temperature 0, 256 output tokens); encoder Qwen3-Embedding-8B; judge DeepSeek-V4-Flash; 128 KiB cap, 4,096 readout tokens |
| Procgen | Shared recurrent PPO controller, 5.0M frames over 16 games | Best development return, then frozen | 320 units from 16 games; disjoint training, development, and evaluation levels; 64 KiB cap |
In the updated manuscript, Table 4 (Appendix E) gives these training budgets, stopping rules, and settings.
Thanks to the authors for this thorough response.
I appreciate the comparison to more baselines, which helps to more clearly illustrate the benefit of the proposed method (e.g., on Alchemy). I also appreciate the clarification of eval tiers, etc.
On ProcGen, a few points:
- It's pretty concerning that return is falling with more frames of training? That makes all the procgen results more suspect, since it suggests something is wrong with the training setup. The training is also still substantially shorter than the original paper.
- It's nice to see CIs that show the proposed method significantly improves over no memory, but that's a bit of a weak baseline. The more appropriate comparison would be whether the proposed method significantly improves over the prior baselines (as is reported e.g. in Table 6 for LongMem), which I can't directly tell from the reported results, but eyeballing it looks like it would not?
- AFAICT the paper still doesn't say whether it used the easy or hard procgen setup (the revisions only address the issues like LongMem tier that I explicitly asked about).
On LongMemV2:
- The performance still seems suspiciously weak, with no method improving more than 5 percentage points over the no-mem baseline (by contrast, the original paper got 40-50% with Qwen3.5-9B using RAG, so there is clearly much more headroom for improvement with an effective memory system).
I have increased my rating to a 2
Response to Reviewer Yhf7 (1/2)
We sincerely thank you for your time, detailed feedback, and recognition of the novelty of the mechanism and the breadth of the validation. Your careful questions on the evaluation, from whether the models are trained enough to how RPC compares with simple references, showed us that our setup needs to be stated more clearly.
Below, we address your two main concerns in turn. On whether the results are low, the Alchemy returns are high on the scale of our setting: RPC reaches 86% of the best achievable return and clearly exceeds the simple heuristic. On whether the models are trained enough, memory is written online rather than trained, and training the Procgen controller 16 to 32 times longer leaves RPC’s lead intact.
Yes. RPC keeps the lowest visual prediction error and the highest mean return among memory methods at every training stage from 4M to 64M frames, and also on single-game Maze controllers trained for 10M frames.
First, we would like to clarify the role of the 4M-frame checkpoint. Our paper evaluates bounded online memory (64 KiB) rather than RL policy optimization, so the lightweight shared controller serves solely as a frozen common visual encoder and base policy that lets all memory mechanisms be compared on identical two-attempt trajectories within each level. What matters is therefore whether this comparison changes as the controller is trained longer.
Second, as you suggested, we reran the evaluation with longer training: shared controllers were trained up to 64M frames, with checkpoints at 16M, 32M, and 64M (4, 8, and 16 times the submitted checkpoint), and single-game Maze controllers to 10M frames (32 times the per-game frames):
Takeaway: RPC has the lowest prediction error of all methods at every stage, cutting the no-memory error by 38% to 51%, and on Maze it beats the strongest baseline, RGMem, on 85.3% of episodes. On closed-loop return, RPC also has the highest mean in every row, at or above no memory. The advantage of RPC therefore persists under 16 to 32 times more training.
As for other tasks, Procgen is the only benchmark with a trained controller, so the results on Alchemy, the controlled suite, and LongMemEval-V2 do not depend on training length. The Alchemy returns are on a smaller scale because of shorter episodes than in Wang et al., as detailed in W2 & Q2.
In the updated manuscript, Section 4.4 and Table 7 (Appendix G) report the extended results from 4M to 64M and on Maze 10M.
RPC improves substantially on the random heuristic: it scores 24.2 points above the official RandomActionBot on the ten paper worlds and 19.1 points above it on 128 worlds, while every memory baseline stays within 4.7 points of the bot.
First, regarding the gap to the optimal return, our returns are on a different scale from Wang et al.: our episodes have 5 trials with 8 potions each, against 10 trials with 12 potions, and episode return grows with both. Measured against the ceiling of our own setting, an oracle that knows the true chemistry (98.8), RPC reaches 86% of the best achievable return, while the random heuristic reaches 61%.
Second, as you suggested, we ran the official RandomActionBot on the same worlds:
Takeaway: Both paired intervals exclude zero. The gain comes from learning the hidden chemistry: RPC shares chemistry across anchors, so it predicts the outcomes of pairs it has not tried. RPC therefore improves well beyond the random heuristic.
In the updated manuscript, Section 4.2 and Table 2 (Appendix D.1) add the RandomActionBot and oracle references, and Appendix D.1 details the comparison with Wang et al.