- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses contextual Markov decision processes (CMDPs) in which the context is fixed within an episode but cannot be observed directly, using the sum of transition embeddings as memory. It motivates this structure through the permutation invariance of the context posterior and proposes STORE: recomputing embeddings for a random subset, reusing the remaining cached embeddings, and training at randomly sampled loss positions. When the cache exactly matches the current encoder, the encoder gradient scaled by T/k is an unbiased estimator of the full-trajectory gradient.
Critical strengths
-
The architecture matches the problem structure. Proposition 4.1 on p. 5 and the theoretical appendix on pp. 14–17 provide explicit arguments for expressivity and gradient estimation. The main value lies in turning existing ideas about permutation-invariant representations into a memory architecture for policy and value networks, while enabling random-subset training to cover the full history.
-
Efficiency and long-term memory are supported by experiments. Results on MuJoCo, Meta-World, and T-Maze cover reward variation, dynamics variation, and long-range memory. Figures 2–4 on pp. 6–8 support MATE’s efficiency and the benefits of STORE. With k=100, STORE still achieves the maximum return on Passive T-Maze at length 800, demonstrating that the training subset length can be separated from the length of usable history. The improvements on Meta-World are also more convincing than results restricted to already saturated tasks.
Critical weaknesses
-
Performance and timing protocols are not fully aligned. Table 3 on p. 18 indicates that the GPT-2 performance results on Hopper and Walker use one layer, whereas Section B.3 on p. 19 states that Table 2 times a two-layer GPT-2. Full-run training times for Mamba and STORE are estimated through cross-device calibration, with a calibration error of approximately 20%; they are not all measured directly under a common setup. The “Measured” caption of Table 2 on p. 6 should explicitly identify these differences. Small gaps, such as 4.2 versus 4.5 hours on Cheetah-Vel, are insufficient to support a robust advantage on every task.
-
Applicability is evaluated only in environments selected or adapted to fit the CMDP assumptions. Appendix C (pp. 20-22) describes settings with a fixed context within each episode and observations intended to be Markov given that context. In particular, T-Maze is explicitly modified to satisfy the CMDP formulation. These choices limit the evidence for applicability to more general environments, where hidden state or temporal dependencies may not be captured by a fixed context. The authors should clarify the scope of applicability and how much the reported performance depends on these environment choices and adaptations.
Questions for authors
- On representative long-memory tasks, how do STORE’s benefits change with exact caches or more frequent refreshes? Do the same advantages hold when Mamba or SplAgger is included and final or sustained performance is reported?
- Are there existing third-party benchmark environments on which MATE can be evaluated directly, without modifying their observations, rewards, or dynamics to satisfy the CMDP assumptions?
2 comments
Table R1 Update
Dear Reviewer 5szF,
We have finished 2 more seeds and updated Table R1 to 5 seeds. As before, less frequent refreshes (i.e., inexact embeddings) can sometimes perform better. Unlike before, STORE is no longer the best at and , and the results show a more consistent trend where more frequent refreshes become optimal as grows.
Table R1 (Updated). STORE () with different refresh rates. Maximum return on Active T-Maze, mean over 5 seeds. refresh re-embeds transitions per update but backpropagates through only .
| Variant | |||||
|---|---|---|---|---|---|
| 1× refresh (STORE, Fig. 3) | 1.00 | 1.00 | 0.89 | 0.80 | 0.54 |
| 2× refresh | 1.00 | 1.00 | 0.92 | 0.78 | 0.64 |
| 4× refresh | 1.00 | 1.00 | 0.86 | 0.83 | 0.71 |
| × refresh (exact cache) | 1.00 | 1.00 | 0.88 | 0.76 | 0.72 |
We hope this update is helpful, and we are happy to clarify anything further.
Dear Reviewer 5szF,
Thank you for recognizing the strengths of MATE and STORE, and for catching the gaps in our timing protocol.
We aligned the timing protocol with Table 3 (one-layer GPT-2 on Hopper-Param and Walker-Param), measured all models under a common setup, and updated Table 2 in the revised PDF. MATE is now 1.10–1.75× faster than every memory baseline on every task, far beyond the s.d. over seeds (≤2.1%). On Cheetah-Vel, 4.2 vs. 4.5 h becomes 0.87 vs. 0.98 h.
The common setup is a single RTX 5090 that runs one model at a time, with the full training time extrapolated from its rate after warm-up (App. B.3). The original Table 2 instead timed full runs, which take up to ten days on V100s, so it had to combine cross-device results by calibration. The extrapolation avoids this calibration error.
We respectfully note that the CMDP setting is broad rather than narrow. It is the standard formulation of meta-RL, where a family of MDPs is parameterized by a context (Beck et al., 2025). The same structure arises in real-world problems such as sim-to-real transfer, healthcare, and autonomous driving, where physical parameters, patients, and road surfaces act as contexts governing the dynamics (Kirk et al., 2023).
Many existing meta-RL benchmarks are CMDPs by construction. All benchmarks in Fig. 2 needed no changes to satisfy the CMDP assumptions. T-Maze, in contrast, is not a meta-RL benchmark but a memory benchmark included to test the memory horizon, so casting it as a CMDP required modifications.
On a long-memory task (Table R1), STORE surprisingly matches or outperforms the exact cache and 2× and 4× more frequent refreshes at every L except 1000. We see the same trend on Meta-World and conjecture that the inexact embeddings regularize training (see our response to Reviewer 3cw7, Q2). The exception at L=1000 suggests that the refresh ratio k/T should not be too low. We are running 2 more seeds to confirm these results and will update Table R1.
Table R1. STORE (k=100) with different refresh rates. Maximum return on Active T-Maze, mean over 3 seeds. r× refresh re-embeds rk transitions per update but backpropagates through only k.
We ran Mamba and SplAgger on T-Maze after the submission (Table R2). On the unsaturated Active T-Maze, STORE performs best at L≤400 but falls behind Mamba at L≥600. The final return (italic in Table R2) shows the same trends. Our response to Reviewer 3cw7 (Q3) explains why the two returns agree.
STORE likely falls behind at large L because the key information in T-Maze comes at the first step: MATE weighs all transitions equally, so the cue makes up only about 1/T of the memory. Mamba can avoid this by learning to keep the cue in its hidden state and ignore the uninformative corridor steps.
The main advantage of STORE, however, is that it can learn dependencies longer than the subset size k (up to L=8k on Passive T-Maze). Contiguous-window training, the default in prior work, cannot go beyond k: GPT-2 fails to learn dependencies even one step longer than k (Fig. 8).
Table R2. Maximum (final) return on Passive / Active T-Maze, mean over 5 seeds. Final return: mean over the last 10% of training. Bold: best maximum; italic: best final.
We are glad to clarify any remaining concerns and hope our responses are reflected in your final assessment.
Best regards, The Authors