- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Context-based offline meta-RL learns universal policies conditioned on task contexts from static datasets, yet existing methods fail severely under context shift due to limited task representations from fixed behavior-policy data. Inspired by biological sleep consolidation, this work introduces MADE, a memory-augmented framework that actively expands the latent task distribution. MADE uses an uncertainty-aware memory bank, a counterfactual task generator with interpolation and conservative extrapolation, and a meta-critic to filter synthetic trajectories. Across five benchmarks, MADE improves OOD performance by up to 13.10% over prior SOTA and retains up to 94.36% performance, validating active latent distribution expansion.
Critical strengths
- Novel paradigm for offline meta-RL:Drawing inspiration from biological memory consolidation, MADE proposes an active latent-space task expansion strategy to enrich task representations, addressing context shift in context-based offline meta-RL.
- Modular three-component design:The framework combines an uncertainty-aware memory bank, counterfactual task generator, and meta-critic filter. The meta-critic assesses synthetic trajectories from consistency, diversity and causality dimensions to reject low-quality generated samples.
- Strong empirical results:Evaluated over five meta-RL benchmarks. MADE achieves up to 13.10% OOD performance gain against SOTA baselines and maintains up to 94.36% performance retention under distribution shift.
Critical weaknesses
- During autoregressive rollout for synthetic contexts, actions are sampled uniformly over Action Space. But the decoder is trained exclusively on actions from offline behavior policies and rarely observes uniform random actions. This creates severe covariate shift. Meta-Critic only filters after full trajectory generation and cannot detect bad single-step transitions inside accepted trajectories.
- The paper omits the training procedure for decoder . Although the proposed method builds upon UNICORN, we recommend explicitly stating its training protocol and loss function; otherwise, the manuscript remains incomplete
Questions for authors
none
We sincerely thank you for recognizing MADE’s novel task-expansion paradigm, modular design, and strong empirical results. We address your concerns below.
Weaknesses
1. Action distribution shift and generation quality.
We agree that action distribution shift deserves attention, but uniform sampling alone does not establish that severe generation errors necessarily occur. We train the decoder on complete offline SAC replay buffers, spanning early random exploration through near-expert behavior.
Meta-Critic computes state and reward anomaly terms at each timestep, so abrupt state changes or abnormal reward magnitudes reduce the acceptance score. On Ant-Dir in Table 6, accepted samples have rollout divergence of 7.3–8.1, close to the real-data reconstruction baseline of 7.0±0.6 and below the rejected samples’ 11.2±1.4, providing direct evidence of overall quality after filtering. Furthermore, synthetic transitions are used only for representation consolidation and do not enter SAC actor/critic updates. We will clarify the distinction between local anomaly scoring and per-step accuracy guarantees to avoid overstating the filter’s capabilities.
2. Decoder training details.
We agree with your suggestion to specify the decoder’s training procedure and loss. Our paper introduces the supervised decoder loss and sleep-consolidation objective, with corresponding configurations in Table 14. We will provide further details.
During the wake phase, the encoder and decoder are jointly trained on real offline data. Given z=qϕ(c), two conditional Gaussian prediction heads predict state increments and rewards. The supervised loss is
Ldec=−ED[logpωΔ(s′−s∣s,a,z)+logpωr(r∣s,a,z)].
During the sleep phase, accepted synthetic contexts further consolidate the encoder and decoder through latent-consistency and decoder-reconstruction losses. We will add this loss and update procedure to the main text so that the training protocol can be understood without consulting UNICORN.