acceptodds
Under review as a conference paper at ICLR 2027

MADE: A Memory-Augmented Distribution Expansion Approach via Sleep Consolidation for Context-Based Offline Meta-Reinforcement Learning

Abstract

Context-based offline meta-reinforcement learning acquires universal policies from fixed datasets by conditioning on task contexts, thereby enhancing agent generalization to unseen tasks. However, existing methods suffer from severe performance degradation under context shift, where test-time contexts are collected by behavior policies that differ from those seen during training. This brittleness arises because prior approaches optimize task representations solely on training data collected by fixed behavior policies, leaving agents vulnerable to out-of-distribution (OOD) failures at test time. Inspired by sleep consolidation in mammalian brains, we propose MADE, a memory-augmented framework that actively expands the task distribution in latent space during offline training, thereby enriching the diversity of context representations. MADE introduces three components: an uncertainty-aware memory bank identifying high-variance regions as expansion priorities, a counterfactual task generator synthesizing novel tasks via interpolation and conservative extrapolation, and a meta-critic filtering synthetic data through consistency, diversity, and causality assessments. Evaluations across five benchmarks show that MADE improves OOD performance by up to 13.10% compared to state-of-the-art methods and achieves up to 94.36% performance retention, effectively validating this active distribution expansion paradigm.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Offline Meta-Reinforcement Learning, Context Shift, Latent Distribution Expansion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iakDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Context-based offline meta-RL learns universal policies conditioned on task contexts from static datasets, yet existing methods fail severely under context shift due to limited task representations from fixed behavior-policy data. Inspired by biological sleep consolidation, this work introduces MADE, a memory-augmented framework that actively expands the latent task distribution. MADE uses an uncertainty-aware memory bank, a counterfactual task generator with interpolation and conservative extrapolation, and a meta-critic to filter synthetic trajectories. Across five benchmarks, MADE improves OOD performance by up to 13.10% over prior SOTA and retains up to 94.36% performance, validating active latent distribution expansion.

Critical strengths
  1. Novel paradigm for offline meta-RL:Drawing inspiration from biological memory consolidation, MADE proposes an active latent-space task expansion strategy to enrich task representations, addressing context shift in context-based offline meta-RL.
  2. Modular three-component design:The framework combines an uncertainty-aware memory bank, counterfactual task generator, and meta-critic filter. The meta-critic assesses synthetic trajectories from consistency, diversity and causality dimensions to reject low-quality generated samples.
  3. Strong empirical results:Evaluated over five meta-RL benchmarks. MADE achieves up to 13.10% OOD performance gain against SOTA baselines and maintains up to 94.36% performance retention under distribution shift.
Critical weaknesses
  1. During autoregressive rollout for synthetic contexts, actions are sampled uniformly over Action Space. But the decoder pθ{p_\theta} is trained exclusively on actions from offline behavior policies and rarely observes uniform random actions. This creates severe covariate shift. Meta-Critic only filters after full trajectory generation and cannot detect bad single-step transitions inside accepted trajectories.
  2. The paper omits the training procedure for decoder pθ{p_\theta}. Although the proposed method builds upon UNICORN, we recommend explicitly stating its training protocol and loss function; otherwise, the manuscript remains incomplete
Questions for authors

none

1 comment

We sincerely thank you for recognizing MADE’s novel task-expansion paradigm, modular design, and strong empirical results. We address your concerns below.

Weaknesses

1. Action distribution shift and generation quality.

We agree that action distribution shift deserves attention, but uniform sampling alone does not establish that severe generation errors necessarily occur. We train the decoder on complete offline SAC replay buffers, spanning early random exploration through near-expert behavior.

Meta-Critic computes state and reward anomaly terms at each timestep, so abrupt state changes or abnormal reward magnitudes reduce the acceptance score. On Ant-Dir in Table 6, accepted samples have rollout divergence of 7.3–8.1, close to the real-data reconstruction baseline of 7.0±0.6 and below the rejected samples’ 11.2±1.4, providing direct evidence of overall quality after filtering. Furthermore, synthetic transitions are used only for representation consolidation and do not enter SAC actor/critic updates. We will clarify the distinction between local anomaly scoring and per-step accuracy guarantees to avoid overstating the filter’s capabilities.

2. Decoder training details.

We agree with your suggestion to specify the decoder’s training procedure and loss. Our paper introduces the supervised decoder loss and sleep-consolidation objective, with corresponding configurations in Table 14. We will provide further details.

During the wake phase, the encoder and decoder are jointly trained on real offline data. Given z=qϕ(c)z=q_{\phi}(c), two conditional Gaussian prediction heads predict state increments and rewards. The supervised loss is

Ldec=−ED[log⁡pωΔ(s′−s∣s,a,z)+log⁡pωr(r∣s,a,z)].\mathcal{L}_{\mathrm{dec}}=-\mathbb{E}_{\mathcal{D}}\left[\log p_{\omega}^{\Delta}(s'-s\mid s,a,z)+\log p_{\omega}^{r}(r\mid s,a,z)\right].

During the sleep phase, accepted synthetic contexts further consolidate the encoder and decoder through latent-consistency and decoder-reconstruction losses. We will add this loss and update procedure to the main text so that the training protocol can be understood without consulting UNICORN.

Reviewer_DJjzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method to handle the OOD problem of context-based offline meta reinforcement learning. This means that the test tasks may be out of the distribution of tasks learned from the training tasks. The idea is to maintain an uncertainty-aware memory bank. The bank is expanded through two strategies: one is Stochastic Manifold Interpolation; the other is Frontier Extrapolation. The two strategies could generate new latent tasks for learning. The experiments on the standard MuJoCo have shown promising results.

Critical strengths

The targeted problem is practical and interesting. The whole paper is well written and easy to follow. The idea is simple and easy to implement. The idea of expanding the 'bank/task distribution' for the OOD issue is reasonable.

Critical weaknesses

The main concern is the experiments. Firstly, the results do not seem statistically significant. For example, in Table 1, UNICORN and MADE are with 330.35+11.12 and 334.51+12.31 on ANT-DIR. Considering the large std, the two methods are not statistically different.

Also, there are some papers in this field that have not been compared, e.g., https://arxiv.org/abs/2008.02790 and https://arxiv.org/abs/2311.06673. The proposed method is more like an 'exploration' method for the task space to improve task generalisation. The authors should find some related works in this direction to compare.

Questions for authors

Can you please add two-sample tests for the results?

Can this method be applied to all offline context meta-RL algorithms?

Why is IID interpolation required if we are targeting the OOD problem?

1 comment

We thank the reviewer for recognizing MADE’s intuitive task-distribution expansion, ease of implementation, and clear presentation. We address the concerns below.

Weaknesses

1. Statistical significance of the results.

MADE’s performance improvements have statistical support across multiple environments. On HalfCheetah-Dir and Hopper-Param, MADE improves IID returns over UNICORN from 2179.52±21.73 to 2318.14±33.62 and from 242.95±3.15 to 293.65±7.53, respectively. Two-sided Welch tests based on the four-seed summary statistics in Table 1 yield approximate pp values of 0.000865 and 0.000235, supporting our performance advantages in reward-varying and dynamics-varying environments. Small gaps on Ant-Dir under standard settings also appear in related work. For example, UNICORN’s Table 2 reports Ant-Dir OOD returns of 233±12 for CSRO and 239±16 for UNICORN-SUP.

Under the extreme task-level OOD setting in our Table 3, the mean-return gap between MADE and UNICORN widens, from 249.73±18.54 to 272.68±11.93, a 9.2% improvement.

Our performance conclusions are supported by evidence across multiple environments and evaluation settings. Smaller gains in some environments or settings do not invalidate this overall conclusion.

2. Differences from related methods and additional comparisons.

We agree that comparisons with task-space exploration and generation methods should be strengthened. Our paper discusses and compares representative methods, including CSRO (NeurIPS 2023), ReDA (AAMAS 2024), UNICORN (NeurIPS 2024), and CERTAIN (ICML 2025).

DREAM learns an exploration policy to acquire task-identifying information through online interaction. We expand the task-representation space using fixed offline data to improve task inference from unseen contexts, without collecting new task information online.

MetaDreamer generates trajectories through latent interpolation and a world model for policy training. We use representation variance stored in memory to target expansion regions, combine interpolation with conservative extrapolation, and explicitly filter candidates before consolidation. Synthetic transitions are used only for encoder–decoder consolidation and do not enter SAC actor/critic updates. This makes expansion more targeted and prevents generated transition errors from directly entering value-learning targets.

To evaluate performance under offline conditions, we added MetaDreamer experiments on HalfCheetah-Dir and Hopper-Param using an offline adaptation. We report means and standard deviations over four random seeds.

MethodHalfCheetah-Dir IIDHalfCheetah-Dir OODHopper-Param IIDHopper-Param OOD
MetaDreamer (offline adaptation)2153.12±24.482083.14±26.38258.67±3.94257.34±4.15
MADE2318.14±33.622187.41±38.37293.65±7.53274.02±4.15

MADE achieves higher mean IID and OOD returns in both environments. The component ablations in Table 4 and matched-candidate controls in Table 8 further support memory-guided sampling and filtering, showing that the gains are not simply due to generating more candidates. We will expand the related-work discussion to include DREAM and MetaDreamer.

Questions

1. Can you add two-sample tests?

We performed two-sided Welch two-sample tests comparing MADE with UNICORN using the four-seed summary statistics in Table 1. Independent training seeds are the statistical units (n=4n=4 per method), with a significance level of 0.05. We applied Holm correction separately to the five IID and five OOD environment comparisons. Representative results are provided in Weakness 1, and we will report the complete raw and adjusted pp values in the revised manuscript.

2. Can MADE be applied to other algorithms?

MADE can, in principle, be combined with other COMRL methods, provided their encoder outputs and conditional decoders are appropriately matched. This paper focuses on whether expanding the task distribution using fixed offline data improves generalization under context distribution shift. Systematic integration across different base algorithms is not our primary research objective.

3. Why is IID interpolation needed?

OOD in our setting primarily arises from changes in context-collecting policies, even when task parameters remain within the training range. Interpolation fills gaps between training-task representations to improve coverage for such contexts, while extrapolation extends latent-space boundaries. The two mechanisms are complementary.

Under the same budget of 64 candidates in Table 12, combining interpolation and extrapolation yields a mean Ant-Dir OOD return of 293.53, compared with 290.54 for extrapolation alone, while increasing IID return from 326.65 to 334.51. These results support retaining interpolation to balance OOD generalization and IID performance.

Reviewer_yzmEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MADE addresses context distribution shift in offline meta-reinforcement learning. It combines a memory bank for variance-guided latent sampling, counterfactual context generation through interpolation and extrapolation, and heuristic quality filtering. An intermittent sleep phase consolidates accepted samples through encoder–decoder objectives. Experiments cover five MuJoCo benchmarks and Meta-World Reach, including context-level shift, limited task-level OOD evaluation, and component ablations. MADE achieves higher mean returns than the evaluated baselines without additional training-time environment interaction.

Critical strengths
  1. The paper addresses a relevant robustness problem. Changes in context-collecting policies can undermine offline meta-RL, and the four-seed main comparisons show consistently higher mean returns across the evaluated benchmarks.

  2. The empirical analysis is reasonably comprehensive. Component ablations, matched-candidate augmentation controls, held-out distribution-distance measurements, simulator-based rollout validation, and training-time comparisons provide useful evidence beyond aggregate performance.

Critical weaknesses
  1. The reliability of counterfactual contexts remains insufficiently characterized. Internal consistency and heuristic filtering do not establish accurate dynamics or rewards at interpolated and extrapolated latents. Appendix B.2 provides useful external validation, but the latent-to-simulator parameter mapping and rollout-error specification are insufficiently described, limiting assessment and reproducibility. Removing the filter lowers mean Ant-Dir OOD performance below UNICORN, underscoring the importance of understanding residual generation errors and failure conditions.

  2. The incremental contribution over closely related methods is not sufficiently established. LDM already constructs imaginary tasks through latent mixtures, while TAVT includes task-preserving generation and distribution alignment. MADE should more clearly establish what its memory-guided sampling, filtering, and sleep consolidation add beyond these mechanisms. Comparisons also omit relevant offline methods such as MetaSTAR. TAVT’s different training protocol warrants matched component-level or offline-adapted comparisons rather than direct numerical comparison.

Questions for authors
  1. Do synthetic transitions enter SAC actor/critic updates? Which policy and decoder parameters receive sleep-loss gradients? Are the gains attributed primarily to improved task inference or policy training at expanded latents?

  2. What substantive advance distinguishes MADE from TAVT’s virtual-task construction and task-preserving objective? Can comparisons with relevant adapted components and MetaSTAR substantiate this distinction?

  3. Are other-task behavior policies executed in the target task’s environment when collecting OOD contexts? Please specify the extreme-OOD parameter ranges, separation boundaries, and task-sampling procedure.

  4. How are synthetic latents mapped to simulator task parameters in Appendix B.2? Please specify the rollout horizon, divergence metric, and normalization.

  5. How do candidate counts, accepted samples, and update budgets differ between Tables 4 and 12? Beyond the reported training-time comparison, how does MADE compare with UNICORN under the same wall-clock budget?

1 comment

We thank the reviewer for recognizing MADE’s performance advantages and comprehensive empirical validation. We address the concerns below.

Weaknesses

1. Reliability of counterfactual contexts.

We designed three complementary criteria covering task semantics, distribution matching, and anomalies in state transitions and rewards to filter synthetic data. Our component ablations support the effectiveness of this design, while the error comparisons in Table 6 provide further evidence of its filtering effect. On Ant-Dir, accepted samples have rollout divergence of 7.3–8.1, close to the real-data reconstruction baseline of 7.0±0.6, whereas rejected samples have divergence of 11.2±1.4.

The details of Appendix B.2 are provided in our response to Question 4.

We introduced the filter precisely to prevent harmful samples from entering training as the generator expands the candidate space. Table 4 shows that filtering increases the mean Ant-Dir OOD return from 271.23 to 293.53. This result supports the intended role of the filter.

2. Differences from related methods.

LDM samples latent mixtures to generate imaginary rewards, while TAVT constrains virtual-task generation through task-preserving objectives and distribution alignment. Both original training protocols use environment interaction. MADE uses fixed offline data to prioritize high-variance latent regions, filter generated candidates, and periodically consolidate accepted contexts, making expansion more targeted and reducing the influence of generation errors on representation learning. Tables 4 and 8 show higher returns for full MADE than the component ablations, Latent Mixup, and random pairing, supporting targeted expansion and filtering.

We also evaluated MetaSTAR on HalfCheetah-Dir and Hopper-Param, covering reward and dynamics variation, respectively. We ran four random seeds and report mean ± standard deviation.

MethodHalfCheetah-Dir IIDHalfCheetah-Dir OODHopper-Param IIDHopper-Param OOD
MetaSTAR2104.44±25.411965.59±37.13256.67±6.41243.01±8.89
MADE2318.14±33.622187.41±38.37293.65±7.53274.02±4.15

MADE improves OOD returns over MetaSTAR by 11.29% and 12.76%, respectively, supporting its performance advantage in both environments.

Questions

1. How are synthetic transitions used?

Synthetic transitions do not enter SAC actor/critic updates and are used only for sleep consolidation. The latent-consistency term updates the task encoder, while the reconstruction term updates the decoder. We attribute the gains primarily to improved task inference and broader latent coverage through expansion of the task-representation distribution.

2. How does MADE differ from other methods?

Please see our response to Weakness 2.

3. How are OOD contexts collected?

Under the proposed extreme task-level OOD protocol, we execute behavior policies from different tasks and training stages in target-task environments whose parameters lie outside the training range. Each environment is split according to its task parameters. For Ant-Dir, training directions lie in [0.6π,1.4π][0.6\pi,1.4\pi], while extreme-OOD directions lie in [0,0.4π]∪[1.6π,2π)[0,0.4\pi]\cup[1.6\pi,2\pi), leaving a 0.2π0.2\pi (36∘36^\circ) gap on each side. We uniformly sample 20 training tasks and 10 test tasks from each OOD interval, use a fixed sampling seed, and share the same task split across all compared methods.

4. How are synthetic latents mapped to simulator parameters?

In Appendix B.2, we fit a ridge-regularized affine map (λ=10−3\lambda=10^{-3}) from training-task latent prototypes to known simulator parameters, using sine/cosine coordinates for angles and log coordinates for positive physical parameters, and report separate state/reward RMSE over 10-step autoregressive rollouts with identical initial states and actions, standardizing errors by per-variable offline-training standard deviations floored at 10−310^{-3}.

5. How do sample and compute budgets compare?

Table 4 ablates components from the default 32+32 candidate setup, whereas Table 12 fixes 64 candidates and varies the interpolation/extrapolation ratio. Acceptance uses the same quality threshold of 0.5, with all candidates accepted when filtering is removed. Both use 200k SAC updates, plus 50 consolidation updates every 50 iterations when enabled.

In comparisons at equal training-time budgets, MADE achieves 7.25% and 9.05% higher test returns than UNICORN on HalfCheetah-Dir (7.36 hours) and Walker-Param (6.79 hours), respectively.

Comments on the paper

1 comment

We thank the Area Chair and all three reviewers for their feedback. We summarize the main concerns and our responses below. 

Reviewer yzmE 

  • Are the generated contexts reliable? Table 6 shows lower rollout error for accepted samples than for rejected samples. In our response, we specified the mapping from latents to simulator parameters, the 10-step rollout metric, and the extreme OOD split for Ant-Dir. Accepted synthetic transitions consolidate the encoder and decoder and do not enter SAC actor or critic updates.
  • What does MADE add beyond prior methods? Unlike LDM’s latent mixtures and TAVT’s task-preserving objectives, MADE uses memory to target where to expand and filters which contexts to consolidate using fixed offline data. Tables 4 and 8 support the roles of filtering and targeted expansion. In new four-seed comparisons, MADE achieved 11.29% and 12.76% higher mean OOD returns than MetaSTAR on HalfCheetah-Dir and Hopper-Param, respectively.
  • Are the sample and compute budgets comparable? We clarified the candidate counts, acceptance rules, and update budgets in Tables 4 and 12. In a separate single-seed comparison at matched training times, MADE achieved 7.25% and 9.05% higher test returns than UNICORN on HalfCheetah-Dir and Walker-Param, respectively.  Reviewer DJjz 
  • Are the gains statistically supported? We ran two-sided Welch tests against UNICORN using the four-seed results in Table 1 and applied Holm correction separately to the IID and OOD comparisons. The IID gains on HalfCheetah-Dir and Hopper-Param remain significant after correction (adjusted p = 0.00346 and 0.00117, respectively). The reviewer noted a small gap on Ant-Dir in the standard IID setting. Table 3 shows that MADE’s mean return advantage over UNICORN widens to 9.2% under extreme task-level OOD.
  • Are task exploration baselines missing? We explained why DREAM’s online exploration uses a different training setup and added four-seed comparisons with an offline adaptation of MetaDreamer. MADE achieved higher mean IID and OOD returns on both HalfCheetah-Dir and Hopper-Param.
  • Why interpolate, and can MADE work with other algorithms? Interpolation fills gaps between training-task representations, which matters when contexts shift even within the training task range. With the same candidate count, combining interpolation and extrapolation gave higher mean Ant-Dir IID and OOD returns than extrapolation alone (Table 12). Integration with other algorithms requires a compatible task encoder and conditional decoder.  Reviewer iakD 
  • Does uniform action sampling compromise generation quality? The decoder’s training data span early exploration through near-expert behavior. The filter scores state and reward anomalies at each timestep and aggregates them across the rollout. Table 6 shows lower overall rollout error for accepted samples.
  • How is the decoder trained? We provided the supervised decoder loss on real offline data and explained how accepted synthetic contexts update the encoder and decoder during sleep.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.