acceptodds
Under review as a conference paper at ICLR 2027

METTA: Memory-Enhanced Test-Time Adaptation for Meta-Reinforcement Learning

Abstract

Meta-reinforcement learning aims to train agents that adapt quickly to unseen tasks from limited interaction, but this requires representing task evidence over long interaction histories. Classical recurrent agents compress experience into hidden states, while transformer-based agents retrieve from past trajectories at quadratic memory and computation cost. In this work, we propose Memory-Enhanced Test-Time Adaptation (METTA) for meta-reinforcement learning. It decomposes the fast adaptation phase into two temporal scales: a causal neural memory performs fine-grained within-episode adaptation with linear sequence complexity, while compact episode-level summaries support long-range cross-episode task inference without attending over the full interaction history. Moreover, equipping agents with neural memory provides sufficient expressive capacity and test-time training capability, enabling effective adaptation to unseen held-out tasks and task families at meta-test time. On unseen held-out Meta-World ML10/45 test environments, METTA achieves a success rate over while maintaining a low training budget and short wall-clock time. On long-context sparse-reward PointMaze tasks, it outperforms transformer-based baselines by exhibiting more structured exploration behavior. These results show that METTA provides an efficient path toward more adaptive and scalable long-context meta-RL.

Primary area:
reinforcement learning
Keywords:
Meta-Reinforcement Learning, Test-Time Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GzBdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on using an efficient memory mechanism within in-context meta-RL. The authors suggest to decompose the adaptation into “within-episode” context encoding and cross-episode memory aggregation. First, a “Test-Time Training” neural memory module is used for the within-episode stage,, which is then aggregated via cross-attention or EMA. The authors evaluate against competitive meta-rl baselines and test on standard meta-rl benchmarks (such as Meta-World). They report improved sample efficiency, wall-clock speed and overall return.

Critical strengths
  1. Meta-RL approaches indeed suffer from severe inefficiency due to very long meta-episodes, which require effective memory for proper adaptation. So I find the problem setup well motivated and worth investigating. It also makes sense to tackle this problem with explicit memory mechanisms and hierarchical formulation (within-episode and cross-episode adaptation).
  2. The qualitative analysis, such as the t-SNE visualization across training are very informative
Critical weaknesses
  1. [Critical] Problematic Evaluation Setup: The reported performance of the baselines is very low, even in widely used and simple benchmarks such as HalfCheetahVel and PointMaze. This raises concern about hyperparameter tuning and chosen evaluation setup. For example, on HalfCheetahVel, the reported return of VariBAD and PEARL in the original papers are higher than those reported in the paper. If the reason for these low returns is the small meta-training budget, I would expect the authors to still show the full training curves and focus on sample efficiency. Currently the results seem misleading (appears like no baseline is able to solve the PointMaze environment).
  2. [Critical] Limited Novelty: Combining two temporal levels in meta-rl (episodic and cross-episodic) for improved efficiency has been studied before (e.g ECET). Fundamentally, METTA replaces the within-episode transformer of ECET with an off-the-shelf TTT. Further, in Figure 9 in the Appendix, the authors show that replacing the TTT with a linear transformer doesn’t deteriorate results. Figure 10 further shows that replacing the attention mechanism with the EMA also does not affect results. These two figures, together with the conceptual similarity to prior works, and the previous point on the evaluation setup, puts the novelty and impact of this work in question.
  3. [Minor] Missing Limitations Discussion: The paper contains no limitations section about the usage of neural memory for meta-rl.
Questions for authors

No further questions.

4 comments

We thank the reviewer for the careful evaluation and the opportunity to clarify several points regarding our experimental setup and contributions.

W1. Evaluation and Baseline Performance

Baseline Implementations and Hyperparameter Tuning. We used the ECET repository (ICLR 2025), which is an adapted version of the TrMRL codebase, as the basis for our baseline implementations, retaining the architectures and evaluation setups of the respective methods. For our main competitors, ECET and TrMRL, we followed their original implementations and hyperparameter configurations [23,34]. Notably, our TrMRL results are even better than those reported in previous work [34]. For the remaining baselines, we used the default hyperparameters provided in the ECET codebase [34] where available. Otherwise, we tuned their hyperparameters on the standard HalfCheetahVel benchmark and subsequently adjusted settings such as batch size and the number of parallel environments for Meta-World. Consequently, PEARL, VariBAD, and RL^2 also achieve competitive, and in some cases better, results than previously reported [34].

One important difference between our evaluation protocol and ECET's is that ECET uses separate exploration and test episodes (For example, 25 episodes for each in Meta-World). Therefore, their reported test performance follows 25 exploration (warm-up) episodes, whereas our evaluation measures adaptation directly from the first episode. This difference should be considered when comparing our reproduced ECET results with those reported in their paper. Since our goal is to evaluate fast adaptation, warm-up episodes would favor methods that require more interactions to adapt.

HalfCheetahVel. Figure 3(a, left) shows the meta-training returns, whereas Figure 3(a, right) shows the adaptation performance across meta-test episodes, indicating whether the methods have learned to adapt to different target velocities. As shown in the episode-wise results, PEARL and VariBAD achieve performance similar to that reported in previous work [23,29,34,47]. RL^2 performance is likewise consistent with previous evaluations [23,34]. Finally, ECET and TrMRL use approximately 4× and 20× our training budget, respectively, and their gradual improvement is consistent with previous results [23,34]. An additional implementation difference is that the original baselines used the older Gym and mujoco-py implementations, whereas our experiments use Gymnasium and MuJoCo (our HalfCheetah-v4 observation has 17 dimensions, compared with 20 in the older implementation, which includes three additional center-of-mass features).

Sparse-reward PointMaze. PointMaze should not be mistaken for an easy-to-solve benchmark in our setting. We designed this sparse-reward PointMaze configuration to investigate long-context exploration under sparse feedback and cross-episode information reuse. Although PointMaze would be considerably easier with dense rewards, our setting provides reward only upon reaching the goal, with a 500-step horizon (Appendix C.3; Table 2). Its long horizon and sparse rewards make long-context memory essential (Section 4.2.1, lines 357 before edit). Regarding baseline performance, Figure 3(b) shows that the baselines exhibit limited improvement in success rate over the evaluated training budget, both during training and testing, particularly the short-context transformer-based methods (Lines 359 before edit). Our illustrative examples (Figures 1; Appendix G.6) and quantitative results (Table 1) further show that these methods repeatedly revisit previously explored regions rather than effectively reusing information across episodes.

Note that Figure 4 uses a wider y-axis range than ECET [34], making the baselines appear closer to zero due to the larger performance gap. Finally, our main comparison uses a budget of 1e7 meta-training steps for ML10, whereas ECET uses 5e7 [34]. Even when considering results up to 5e7 meta-training steps, the best-performing baseline remains at least 20% below METTA, based on previously reported learning curves [23, 34]. Moreover, the baseline training success in ECET remain below 20% during the first 1e7 meta-training steps, as shown in their learning curves [34].

W2. Novelty and Contributions

We would like to further clarify the distinction between METTA and ECET. While both methods share the high-level idea of operating at two temporal scales, METTA differs fundamentally in how both within-episode adaptation and cross-episode aggregation are implemented and handled, during both training and meta-testing. In addition, our implementation uses sequential PPO training, processing episodes chronologically and reconstructing the required causal history, rather than independently sampling transitions and their contexts as in ECET. In the following paragraphs, we discuss how these design choices work together to enable efficient long-context adaptation and examine their respective contributions.

W2. (continued)

A major limitation of ECET is its substantial computational and memory overhead (Section 4.4 line 464 before edit, and Appendix F), which can also limit how effectively it utilizes cross-episode information, as illustrated by our PointMaze results (Figures 1 and Appendix G6). Simply replacing its within-episode Transformer with TTT would further increase its cost due to TTT's constant per-step update overhead. METTA instead co-designs its memory and training procedure (App. B.1–B.3). ECET uses large minibatches and independently samples previous contexts for individual transitions, requiring repeated forward passes and producing numerous context-dependent gradient directions. Moreover, ECET samples only 5 transitions from each previous episode, which can limit the historical evidence available for task inference. This is consistent with our PointMaze results (Figure 1; App. G6), where ECET repeatedly revisits previously explored regions.

In contrast, we use sequential PPO optimization, processing episodes chronologically and reconstructing only the required causal history. Note that applying METTA's training procedure with a Transformer architecture of O(N^2) complexity would be computationally impractical in the long-context settings we target. This avoids excessive forward passes and roughly halves PPO update time (Appendix B.1). For 1e7 ML10 interactions, METTA requires approximately 20K PPO backward passes, compared with approximately 3.1M backward passes in ECET. Although these counts reflect different update structures, they illustrate the substantial difference in computational workload. This is also reflected in Figure 7, where METTA achieves stronger performance with substantially lower GPU-hour cost. Furthermore, our method supports random subsampling of previous episodes to bound the training context, and uses a relatively large TTT chunk size of 128 (Appendix B.2), balancing parallel gradient computation with temporal dependency preservation.

We would also like to clarify a potentially misleading interpretation of Figure 9 (Appendix G.1). The linear Transformer does not resemble ECET; it is another variant of METTA, where we replace only the within-episode TTT memory while keeping the rest of the framework unchanged. In fact, we already introduce the linear Transformer in the preliminaries (Section 2.2) as a simple example of neural memory, where linear attention can be interpreted as a recurrent memory with an internal matrix-valued state. Thus, this experiment investigates the flexibility of our memory framework (replacing the TTT module with a simple linear attention) rather than comparing against ECET. Although this variant performs slightly worse and fluctuates more than the default version of METTA with TTT, its competitive performance demonstrates that METTA can achieve nearly comparable results with a lower-complexity memory mechanism, demonstrating that the benefits of METTA are not restricted to a particular neural memory architecture.

Similarly, EMA is not introduced merely as a competing method. As described in Section 3.2 (line 253 before edit), we proposed a cheaper alternative to cross-episode attention using a simple context averaging mechanism, rather than Transformer blocks as in ECET. Although its performance gap is noticeable on PointMaze, it remains competitive in several settings while providing more efficient aggregation (Appendix G.2, Figure 10). We include EMA as a simple aggregation baseline within METTA, providing a practical alternative to attention with a different performance-efficiency trade-off.

Finally, our ablations reveal a potential issue with existing meta-RL benchmarks that, to our knowledge, has not been sufficiently investigated in previous work. Figure 6 (Section 4.3) shows that disabling online TTT updates has little effect on ML10, where goal coordinates are observed (like multi-task learning) and the task families are sufficiently distinct to be identified from relatively few interactions, as also illustrated by the t-SNE visualization. In contrast, on PointMaze and hidden-goal ML1 Push, the goal position (task) must be inferred through interaction, making task inference substantially harder and test-time adaptation more important. This is not a drawback of METTA; rather, our ablation separates the ability to infer which task is being performed from the ability to learn how to perform it. We further introduced sparse PointMaze specifically to expose this distinction in a long-context, sparse-reward setting, where retaining and reusing exploration evidence is essential.

W3. Limitations

We thank the reviewer for pointing this out. We have added a dedicated Limitations section discussing the current scope of METTA. We also discuss several directions for future work, including temporal memory for nonstationary environments, alternative neural memory architectures, RL-specific inner-loop objectives, and off-policy training.

Dear Reviewer,

We would appreciate an opportunity to further discuss/address any concerns that might remain after our rebuttal response. We look forward to hearing from you. Thanks!

Dear Reviewer, We would greatly appreciate your feedback on our rebuttal and welcome the opportunity to discuss any remaining concerns before the discussion deadline.

Reviewer_9f3VOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes METTA, a meta-RL framework combining within-episode test-time training (TTT) memory with cross-episode aggregation of compact episode summaries. The memory undergoes causal self-supervised updates, while the overall agent is trained end-to-end using PPO. This design avoids attention over the complete timestep-level interaction history. Evaluation covers HalfCheetahVel, Meta-World ML1/ML10/ML45, and sparse-reward PointMaze, using five seeds. The reported results show substantial improvements on held-out manipulation tasks and sparse navigation, together with favorable computational costs. Ablations investigate online updates, alternative memory architectures, cross-episode aggregation, memory capacity, and policy conditioning.

Critical strengths
  1. Substantial empirical gains across distinct adaptation settings. METTA achieves large improvements over the evaluated baselines on held-out Meta-World task families and PointMaze (Figures 3–4). Evaluation uses common interaction budgets and five independent seeds. The combination of manipulation generalization and sparse-reward navigation provides meaningful evidence of practical value beyond a single favorable benchmark.

  2. A useful memory design supported by informative ablations. Separating within-episode processing from cross-episode retrieval offers a practical way to retain task evidence without storing the full interaction history. The online-update ablations demonstrate clear benefits on PointMaze and hidden-goal ML1 Push, while the aggregation ablations support the usefulness of retaining previous episode information (Figures 6 and 9–12). These experiments substantiate important components of the proposed framework.

Critical weaknesses

The attribution of task-family generalization to TTT is stronger than the evidence supports. On ML10, disabling online memory updates yields similar performance throughout training and testing. Moreover, the linear Transformer variant reaches comparable final test performance on ML10 and PointMaze . Thus, the results establish the effectiveness of the complete framework more convincingly than a distinctive advantage of gradient-based test-time memory updates. This matters because such updates are central to the paper’s explanation of its generalization gains. Episode-wise ML10/ML45 adaptation curves and evaluation with history reset between episodes would help separate adaptation benefits from transferable policy representations.

Questions for authors

none

1 comment

We sincerely thank the reviewer for the thoughtful review and for recognizing the strong empirical results and informative ablations. We agree that the generalization gains should not be attributed solely to online TTT updates, as our ablation studies also demonstrate the importance of other components of the framework. We address the concerns regarding the role of test-time adaptation in generalization below.

One important thing to note is that the contribution of online TTT updates depends on how task-relevant information is obtained. In ML10, disabling online updates has little effect (Figure 6, Section 4.3), since goal coordinates are observed and task families are sufficiently distinct to be identified from relatively few interactions, as also illustrated by the t-SNE visualization (Figure 5). Thus, the main challenge in ML10 is learning how to perform the task rather than inferring which task is being performed. In contrast, on PointMaze and hidden-goal ML1 Push, the goal position must be inferred through interaction, making task inference substantially harder and online adaptation more important. Disabling online updates causes a significant drop in test performance, particularly on PointMaze. Moreover, Figure 6b shows that TTT improves ML1 Push adaptation from the very first evaluation episode, before any previous-episode information is available. This highlights that cross-episode retrieval alone cannot replace within-episode adaptation when task inference is difficult (Section 4.3, line 431 before edit; Appendix G.3).

Alternative memory architectures and history reset. Regarding the linear Transformer in Figure 9 (Appendix G.1), it is important to clarify that this is not a competing method but another variant of METTA, where we replace only the within-episode TTT memory while keeping the rest of our framework unchanged. In fact, we already introduce the linear Transformer in the preliminaries (Section 2.2, line 154-155 before edit) as a simple example of neural memory, where linear attention can be interpreted as a recurrent memory with an internal matrix-valued state. Thus, this experiment investigates the flexibility of our memory framework rather than comparing it against ECET. Importantly, applying METTA's sequential training procedure with a standard Transformer of O(N^2) complexity would be computationally impractical in the long-context settings we target. We include this variant to demonstrate the flexibility of METTA in accommodating different neural memory architectures. Although the linear Transformer can achieve comparable final performance, its learning curves exhibit more fluctuations and its performance is slightly lower on both ML10 and PointMaze. In particular, TTT consistently maintains higher performance throughout training and testing on PointMaze and this motivates our choice of TTT as our default method over other memory-based methods with linear complexity. We hypothesize that this advantage arises from TTT's surprise-based gradient updates, which allow the memory to adapt its parameters in response to prediction errors and newly encountered task-relevant information, an especially important property in sparse learning signal settings such as PointMaze.

We would also like to point out that Figure 10 (Appendix G.2) already includes a variant without cross-episode aggregation, effectively resetting the cross-episode history between episodes while retaining within-episode TTT adaptation. This ablation isolates the contribution of reusing previous episodes from adaptation within the current episode. Removing cross-episode aggregation substantially reduces test performance on ML10 and delays adaptation on PointMaze, demonstrating the importance of retaining and reusing information across episodes. Together, Figures 6 and 10 distinguish the complementary contributions of within-episode adaptation and cross-episode information reuse.

Reviewer_ADb8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes METTA, a memory-based method for meta-reinforcement learning. It combines a test-time training module for adaptation within each episode with stored episode summaries for using information across episodes. Experiments cover locomotion, robotic manipulation, and navigation tasks. The evaluation includes comparisons with existing meta-RL methods, memory-module ablations, and computational cost analysis.

Critical strengths

The method reports good adaptation results across several benchmarks, supported by comparisons and ablation studies.

Critical weaknesses
  1. The measured scaling with the number of episodes is not clear enough. Section 4.4 and Appendix F already analyze computational complexity, and the EMA variant provides a fixed-size memory alternative. However, Table 4 reports only one latency and memory value for each TTT variant, without clearly specifying the episode count or whether cross-episode attention is included. Measurements of the full attention-based method as the episode count increases would strengthen the practical efficiency evaluation. A discussion of memory and decision-time limits would also help clarify its applicable range beyond the tested settings.
  2. The performance–cost comparison with GRU and EMA variants is incomplete. Figures 9 and 10 provide useful task-performance comparisons, but corresponding inference latency and memory measurements are not reported for these variants. Although the paper includes other computational comparisons, they do not directly establish the trade-off for these particular alternatives. Reporting task performance and computational cost together would help assess the practical value of the default design.
Questions for authors
  1. What episode count is used for METTA in Table 4, and do the reported measurements include cross-episode aggregation? Could the authors show how the full method’s latency and memory usage change with episode count and discuss the implications for hardware constraints?
  2. Could the authors report inference latency and memory usage for the GRU and EMA variants in Figures 9 and 10, alongside task performance under the same hardware and evaluation settings?
1 comment

We sincerely thank the reviewer for the thorough review and for recognizing the strong adaptation results across multiple benchmarks. We have added further measurements and clarified the corresponding evaluation details in Appendix F.

Episode-count scaling. We have clarified the measurement settings in Table 4. All METTA variants are evaluated over E=25 episodes with an episode horizon of H=500, and the reported latency and memory include the complete inference pipeline, including TTT updates and cross-episode aggregation. For TrMRL and ECET, T denotes the sequence-context length, and ECET is evaluated with E=5 and E=25, where E=5 means considering only five previously observed episodes retained for context.

Furthermore, we added Figure 11, which measures the complete inference latency and memory as the number of episodes increases from 1 to 100, with 500 timesteps per episode. The results show that latency remains nearly flat over this range, while memory increases only modestly because the stored episode representations are compact. EMA maintains constant history-dependent memory through a single running summary. Although attention has O(e) per-decision complexity, its measured overhead remains small over the tested range. All measurements were performed on the same system. Even at 100 episodes, latency remains below approximately 2.5 ms per decision and memory below 5.2 MiB.

Computational cost of alternative memories. We extended Table 4 with inference measurements for the METTA-GRU and METTA-EMA variants under the same evaluation settings. METTA with GRU and cross-attention requires 0.349 ms per step and 1.141 MiB, while METTA with TTT-MLP and EMA requires 1.999 ms and 4.769 MiB. The existing METTA (TTT-MLP+CA) and METTA (TTT-Linear+CA) measurements are also retained. Notably, ECET identifies E=25 and T=5 as its best-performing configuration in its ablation study [34]. Even against this configuration, METTA (GRU+CA) achieves approximately 3.0× lower latency and 2.2× lower memory usage (Table 4), while also achieving better task performance than ECET (Figures 4 and 9). Although METTA (GRU+CA) offers substantially lower inference overhead, Figure 9 shows that TTT-based METTA provides stronger test performance, particularly on PointMaze and ML10, highlighting the trade-off between computational efficiency and adaptation capability. Together with the performance results in Figures 9 and 10, these measurements provide a direct comparison of the performance-efficiency trade-offs among different variants of METTA.

Together with Figure 11, these results demonstrate that METTA can scale to longer trials with modest inference overhead under the evaluated hardware and episode-count range.

We hope that these additional experiments and clarifications address all of the reviewer's concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Invitation for Further Discussion

Dear Reviewers,

Thank you again for your thoughtful reviews. We would like to kindly note that, as the discussion deadline approaches, there has not yet been any follow-up engagement. We believe that even brief comments or questions at this stage could help clarify any remaining concerns and contribute to a more complete evaluation of our work.

We sincerely appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.