- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on using an efficient memory mechanism within in-context meta-RL. The authors suggest to decompose the adaptation into “within-episode” context encoding and cross-episode memory aggregation. First, a “Test-Time Training” neural memory module is used for the within-episode stage,, which is then aggregated via cross-attention or EMA. The authors evaluate against competitive meta-rl baselines and test on standard meta-rl benchmarks (such as Meta-World). They report improved sample efficiency, wall-clock speed and overall return.
Critical strengths
- Meta-RL approaches indeed suffer from severe inefficiency due to very long meta-episodes, which require effective memory for proper adaptation. So I find the problem setup well motivated and worth investigating. It also makes sense to tackle this problem with explicit memory mechanisms and hierarchical formulation (within-episode and cross-episode adaptation).
- The qualitative analysis, such as the t-SNE visualization across training are very informative
Critical weaknesses
- [Critical] Problematic Evaluation Setup: The reported performance of the baselines is very low, even in widely used and simple benchmarks such as HalfCheetahVel and PointMaze. This raises concern about hyperparameter tuning and chosen evaluation setup. For example, on HalfCheetahVel, the reported return of VariBAD and PEARL in the original papers are higher than those reported in the paper. If the reason for these low returns is the small meta-training budget, I would expect the authors to still show the full training curves and focus on sample efficiency. Currently the results seem misleading (appears like no baseline is able to solve the PointMaze environment).
- [Critical] Limited Novelty: Combining two temporal levels in meta-rl (episodic and cross-episodic) for improved efficiency has been studied before (e.g ECET). Fundamentally, METTA replaces the within-episode transformer of ECET with an off-the-shelf TTT. Further, in Figure 9 in the Appendix, the authors show that replacing the TTT with a linear transformer doesn’t deteriorate results. Figure 10 further shows that replacing the attention mechanism with the EMA also does not affect results. These two figures, together with the conceptual similarity to prior works, and the previous point on the evaluation setup, puts the novelty and impact of this work in question.
- [Minor] Missing Limitations Discussion: The paper contains no limitations section about the usage of neural memory for meta-rl.
Questions for authors
No further questions.
4 comments
Part 2
W2. (continued)
A major limitation of ECET is its substantial computational and memory overhead (Section 4.4 line 464 before edit, and Appendix F), which can also limit how effectively it utilizes cross-episode information, as illustrated by our PointMaze results (Figures 1 and Appendix G6). Simply replacing its within-episode Transformer with TTT would further increase its cost due to TTT's constant per-step update overhead. METTA instead co-designs its memory and training procedure (App. B.1–B.3). ECET uses large minibatches and independently samples previous contexts for individual transitions, requiring repeated forward passes and producing numerous context-dependent gradient directions. Moreover, ECET samples only 5 transitions from each previous episode, which can limit the historical evidence available for task inference. This is consistent with our PointMaze results (Figure 1; App. G6), where ECET repeatedly revisits previously explored regions.
In contrast, we use sequential PPO optimization, processing episodes chronologically and reconstructing only the required causal history. Note that applying METTA's training procedure with a Transformer architecture of O(N^2) complexity would be computationally impractical in the long-context settings we target. This avoids excessive forward passes and roughly halves PPO update time (Appendix B.1). For 1e7 ML10 interactions, METTA requires approximately 20K PPO backward passes, compared with approximately 3.1M backward passes in ECET. Although these counts reflect different update structures, they illustrate the substantial difference in computational workload. This is also reflected in Figure 7, where METTA achieves stronger performance with substantially lower GPU-hour cost. Furthermore, our method supports random subsampling of previous episodes to bound the training context, and uses a relatively large TTT chunk size of 128 (Appendix B.2), balancing parallel gradient computation with temporal dependency preservation.
We would also like to clarify a potentially misleading interpretation of Figure 9 (Appendix G.1). The linear Transformer does not resemble ECET; it is another variant of METTA, where we replace only the within-episode TTT memory while keeping the rest of the framework unchanged. In fact, we already introduce the linear Transformer in the preliminaries (Section 2.2) as a simple example of neural memory, where linear attention can be interpreted as a recurrent memory with an internal matrix-valued state. Thus, this experiment investigates the flexibility of our memory framework (replacing the TTT module with a simple linear attention) rather than comparing against ECET. Although this variant performs slightly worse and fluctuates more than the default version of METTA with TTT, its competitive performance demonstrates that METTA can achieve nearly comparable results with a lower-complexity memory mechanism, demonstrating that the benefits of METTA are not restricted to a particular neural memory architecture.
Similarly, EMA is not introduced merely as a competing method. As described in Section 3.2 (line 253 before edit), we proposed a cheaper alternative to cross-episode attention using a simple context averaging mechanism, rather than Transformer blocks as in ECET. Although its performance gap is noticeable on PointMaze, it remains competitive in several settings while providing more efficient aggregation (Appendix G.2, Figure 10). We include EMA as a simple aggregation baseline within METTA, providing a practical alternative to attention with a different performance-efficiency trade-off.
Finally, our ablations reveal a potential issue with existing meta-RL benchmarks that, to our knowledge, has not been sufficiently investigated in previous work. Figure 6 (Section 4.3) shows that disabling online TTT updates has little effect on ML10, where goal coordinates are observed (like multi-task learning) and the task families are sufficiently distinct to be identified from relatively few interactions, as also illustrated by the t-SNE visualization. In contrast, on PointMaze and hidden-goal ML1 Push, the goal position (task) must be inferred through interaction, making task inference substantially harder and test-time adaptation more important. This is not a drawback of METTA; rather, our ablation separates the ability to infer which task is being performed from the ability to learn how to perform it. We further introduced sparse PointMaze specifically to expose this distinction in a long-context, sparse-reward setting, where retaining and reusing exploration evidence is essential.
W3. Limitations
We thank the reviewer for pointing this out. We have added a dedicated Limitations section discussing the current scope of METTA. We also discuss several directions for future work, including temporal memory for nonstationary environments, alternative neural memory architectures, RL-specific inner-loop objectives, and off-policy training.
Dear Reviewer,
We would appreciate an opportunity to further discuss/address any concerns that might remain after our rebuttal response. We look forward to hearing from you. Thanks!
Dear Reviewer, We would greatly appreciate your feedback on our rebuttal and welcome the opportunity to discuss any remaining concerns before the discussion deadline.
Part 1
We thank the reviewer for the careful evaluation and the opportunity to clarify several points regarding our experimental setup and contributions.
W1. Evaluation and Baseline Performance
Baseline Implementations and Hyperparameter Tuning. We used the ECET repository (ICLR 2025), which is an adapted version of the TrMRL codebase, as the basis for our baseline implementations, retaining the architectures and evaluation setups of the respective methods. For our main competitors, ECET and TrMRL, we followed their original implementations and hyperparameter configurations [23,34]. Notably, our TrMRL results are even better than those reported in previous work [34]. For the remaining baselines, we used the default hyperparameters provided in the ECET codebase [34] where available. Otherwise, we tuned their hyperparameters on the standard HalfCheetahVel benchmark and subsequently adjusted settings such as batch size and the number of parallel environments for Meta-World. Consequently, PEARL, VariBAD, and RL^2 also achieve competitive, and in some cases better, results than previously reported [34].
One important difference between our evaluation protocol and ECET's is that ECET uses separate exploration and test episodes (For example, 25 episodes for each in Meta-World). Therefore, their reported test performance follows 25 exploration (warm-up) episodes, whereas our evaluation measures adaptation directly from the first episode. This difference should be considered when comparing our reproduced ECET results with those reported in their paper. Since our goal is to evaluate fast adaptation, warm-up episodes would favor methods that require more interactions to adapt.
HalfCheetahVel. Figure 3(a, left) shows the meta-training returns, whereas Figure 3(a, right) shows the adaptation performance across meta-test episodes, indicating whether the methods have learned to adapt to different target velocities. As shown in the episode-wise results, PEARL and VariBAD achieve performance similar to that reported in previous work [23,29,34,47]. RL^2 performance is likewise consistent with previous evaluations [23,34]. Finally, ECET and TrMRL use approximately 4× and 20× our training budget, respectively, and their gradual improvement is consistent with previous results [23,34]. An additional implementation difference is that the original baselines used the older Gym and mujoco-py implementations, whereas our experiments use Gymnasium and MuJoCo (our HalfCheetah-v4 observation has 17 dimensions, compared with 20 in the older implementation, which includes three additional center-of-mass features).
Sparse-reward PointMaze. PointMaze should not be mistaken for an easy-to-solve benchmark in our setting. We designed this sparse-reward PointMaze configuration to investigate long-context exploration under sparse feedback and cross-episode information reuse. Although PointMaze would be considerably easier with dense rewards, our setting provides reward only upon reaching the goal, with a 500-step horizon (Appendix C.3; Table 2). Its long horizon and sparse rewards make long-context memory essential (Section 4.2.1, lines 357 before edit). Regarding baseline performance, Figure 3(b) shows that the baselines exhibit limited improvement in success rate over the evaluated training budget, both during training and testing, particularly the short-context transformer-based methods (Lines 359 before edit). Our illustrative examples (Figures 1; Appendix G.6) and quantitative results (Table 1) further show that these methods repeatedly revisit previously explored regions rather than effectively reusing information across episodes.
Note that Figure 4 uses a wider y-axis range than ECET [34], making the baselines appear closer to zero due to the larger performance gap. Finally, our main comparison uses a budget of 1e7 meta-training steps for ML10, whereas ECET uses 5e7 [34]. Even when considering results up to 5e7 meta-training steps, the best-performing baseline remains at least 20% below METTA, based on previously reported learning curves [23, 34]. Moreover, the baseline training success in ECET remain below 20% during the first 1e7 meta-training steps, as shown in their learning curves [34].
W2. Novelty and Contributions
We would like to further clarify the distinction between METTA and ECET. While both methods share the high-level idea of operating at two temporal scales, METTA differs fundamentally in how both within-episode adaptation and cross-episode aggregation are implemented and handled, during both training and meta-testing. In addition, our implementation uses sequential PPO training, processing episodes chronologically and reconstructing the required causal history, rather than independently sampling transitions and their contexts as in ECET. In the following paragraphs, we discuss how these design choices work together to enable efficient long-context adaptation and examine their respective contributions.