acceptodds
Under review as a conference paper at ICLR 2027

META: Mamba Episodic Temporal Alignment for Stable Reinforcement Learning

Abstract

Structured State Space Models (SSMs), such as Mamba, offer an efficient mechanism for modeling long-range dependencies and are well suited to partially observable reinforcement learning (RL). However, directly applying Mamba to on-policy RL introduces two forms of temporal instability. First, we identify -drift, where the learned temporal scaling parameter , which governs the effective memory duration of the model, gradually diverges during training and destabilizes useful temporal context. Second, we observe episodic contamination, where residual state information persists across episode boundaries and violates episodic independence. To address these issues, we introduce META (Mamba Episodic Temporal Alignment), a framework for stabilizing the temporal dynamics of Mamba in episodic RL. META combines two complementary mechanisms: Episodic Forgetting, which resets temporal traces at episode boundaries to prevent cross-episode interference, and Memory Rebalancing, which regulates the sensitivity of through structured spectral reparameterization to counteract drift during training. Across eight partially observable Brax benchmarks, META improves training stability and sample efficiency over recurrent and SSM-based on-policy baselines in most settings, achieving up to more than higher final returns than the strongest prior SSM baseline, S5RL, on tasks where that baseline attains positive performance. These results highlight temporal stability as a central challenge for deploying selective state-space models in episodic reinforcement learning.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, state space models, Mamba, partially observable reinforcement learning, temporal memory, episodic memory, sample efficiency, sequence modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_U4UcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the instability issue of online RL when mamba is used as the history encoder. It proposes Episodic Forgetting (EF) and Memory Rebalancer (MR) to suppress the information of previous episodes and to mitigate the forgetting problem when the episode is long. The experiments on Brax show the proposed method to have superior or comparable performance with baselines.

Critical strengths

The paper is easy to follow.

Critical weaknesses
  • The methodological novelty is limited. Handling the hidden state reset at the boundary is a basic requirement for online RL. EF itself is not a sufficiently compelling contribution.
  • MR is essentially a simple regularization trick. Its value is hard to evaluate without numerous experiments.
  • The empirical improvements are insufficient. Carefully tuned baselines such as LSTM, transformer, RetNet, should be included. More benchmarks should also be evaluated.
Questions for authors

What is the difference between META and RetNet [1]?

[1] Sun, Yutao, Li Dong, Shao-Han Huang, Shu-Ming Ma, Yuqing Xia, Ji-Long Xue, Jianyong Wang and Furu Wei. “Retentive Network: A Successor to Transformer for Large Language Models.” ArXiv abs/2307.08621 (2023): n. pag.

Reviewer_CAfPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes META, a framework for stabilizing Mamba as a recurrent backbone in episodic on-policy RL. It describes two failure modes, a growth of the temporal-scale parameter ∆ during training and leakage of hidden state across episode boundaries, and addresses them with Episodic Forgetting, which sets the pre-softplus ∆ projection to a constant at episode boundaries as a soft reset, and Memory Rebalancer, which spectrally normalizes the joint input projection that generates ∆ under a learnable gain. Experiments on eight partially observable Brax tasks and controlled memory tasks report improvements over LSTM, Mamba-1 and S5RL baselines and over the Mamba-2 backbone without the two mechanisms.

Critical strengths

S1. The motivation is clear, and the problem of making selective state-space models usable as recurrent backbones in on-policy RL is of practical interest.

S2. Extensive experiments and controls are conducted.

Critical weaknesses

W1. The experiments do not support the main claim. First, the main comparison in Table 1 is unfair: META is built on Mamba-2 while the Mamba baseline is Mamba-1, so part of the reported gain is the change of backbone, and in the matched Mamba-2 comparison (Table 6) the improvement is significant on only a minority of the tasks. Second, the ablation in Table 2 does not show that the two components are complementary, since on most tasks one component alone does better than both. What does Mamba-2 without EF and MR score under the protocol of Table 1?

W2. The ∆-drift that motivates the Memory Rebalancer is not shown to be harmful. The growth of ∆ is shown for one unnamed setting (Figure 2), and nothing links a larger ∆ to lower return; the gains from MR are equally consistent with spectral normalization helping optimization on its own [3, 4]. In the long-delay memory test, MR alone is at chance level (Table 4), which is not discussed.

W3. The novelty is limited. Episodic Forgetting is the boundary reset through a large ∆ already described in the Mamba paper [1], and Memory Rebalancer is spectral normalization of the input projection with a learnable scale [2]; the paper does not say so, and the earlier use of spectral normalization in deep RL [3, 4] is not discussed.

W4. The analysis in Section 3.2 does not support the claim that MR controls ∆-drift. The bound limits the sensitivity of ∆ to the input, not its magnitude, and the non-expansiveness of the transition holds for the unmodified model as well.

Minor comments:

  • Sec. 2.3 and Sec. 3.2 use the Mamba-1 formulation, while the method is implemented on Mamba-2.
  • Tables 3 and 5: the no-reset and EF-off rows switch EF off at inference in policies trained with it, which is not a proper ablation.

References:

[1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces. COLM, 2024.

[2] Spectral Normalization for Generative Adversarial Networks. ICLR, 2018.

[3] Spectral Normalisation for Deep Reinforcement Learning: An Optimisation Perspective. ICML, 2021.

[4] Towards Deeper Deep Reinforcement Learning with Spectral Normalization. NeurIPS, 2021.

[5] Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficient. ICLR, 2025.

Questions for authors

Please refer to the Weaknesses above.

Reviewer_SQcuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies temporal instability when using Mamba as a recurrent backbone for episodic on-policy reinforcement learning. It proposes META, combining Episodic Forgetting (EF), which increases the temporal scale at episode boundaries to attenuate previous-state influence, and a Memory Rebalancer (MR), which spectrally reparameterizes the joint input projection. Evaluation includes partially observable Brax tasks, architecture-matched Mamba-2 ablations, frozen history-swap interventions, controlled memory tasks, and efficiency measurements. The ten-seed matched comparison reports higher mean final returns on seven of eight tasks, with paired confidence intervals excluding zero on three tasks.

Critical strengths

1 Frozen interventions provide evidence that EF reduces cross-episode dependence At fixed policy weights and post-boundary observations, the history-swap experiment reports a 27.3-fold reduction in eight-step policy-logit sensitivity (Table 3). The visual cue-inversion experiment further supports that the successful long-delay policy uses the initial cue rather than a fixed action preference (Section 4.4). These interventions strengthen the evidence beyond return curves alone. 2 The evaluation includes meaningful controls and uncertainty reporting The architecture-matched factorial and ten-seed comparison help separate the proposed modifications from changing the backbone (Tables 2 and 6). The manuscript also includes segmented-reset and direct-clipping controls, and reports approximately 95-97% of vanilla Mamba-2 training throughput in its architecture benchmark. These results support the practical relevance of a lightweight implementation.

Critical weaknesses

1 Delta drift is an incomplete diagnostic of effective memory degradation The paper interprets increasing Δ as evidence of a shortening memory horizon. However, effective decay depends jointly on Δ and the learnable continuous-time state matrix; changes in one can compensate for changes in the other. Consequently, Δ-drift alone does not establish deteriorating memory retention. Furthermore, a short credit-assignment horizon motivates difficulty in learning long-term dependencies, but does not determine the direction of optimization pressure on Δ. The proposed explanation is plausible, yet the causal connection between the diagnosed drift and performance degradation remains insufficiently established. 2 The theoretical properties of MR do not sufficiently address memory collapse The Lipschitz argument characterizes the input sensitivity of the Δ-generation pathway, rather than bounding the absolute temporal scale or guaranteeing a useful memory horizon. The learnable gain and bias can still permit rapid forgetting. Moreover, the non-expansiveness of the state transition also holds for vanilla Mamba with stable diagonal dynamics and allows historical contributions to decay arbitrarily quickly. Thus, the analysis establishes that the linear transition does not amplify the previous state, but does not explain why MR should prevent memory collapse or preserve task-relevant information. 3 Joint projection normalization obscures the temporal specificity of MR MR normalizes the projection generating z, x, B, C, and Δ, simultaneously modifying input writing, state readout, gating, and temporal scaling. This introduces broad optimization and regularization effects, providing several possible explanations for the observed improvements. The method does not sufficiently justify why these functionally different pathways should be normalized jointly, or why the benefits should primarily be attributed to temporal-scale stabilization. This limits the mechanistic interpretation and leaves its specific contribution beyond general spectral normalization unclear. 4 EF couples forgetting with input writing and its approximate reset conditions remain unclear Increasing Δ at episode boundaries attenuates the previous state but also changes the current-input injection. EF therefore modifies how the first observation of a new episode is processed, introducing an additional boundary-dependent bias whose benefits and costs may depend on the task and observation noise. Furthermore, a fixed boundary pre-activation does not guarantee sufficiently strong forgetting as the bias and state matrix evolve during training. A contraction factor strictly below one does not establish negligible cross-episode influence. EF should therefore be characterized as an approximate boundary mechanism with coupled writing and forgetting effects, whose residual dependence and applicability conditions require clearer treatment.

Questions for authors
  1. How is Δ-drift distinguished from changes in the effective decay rates jointly determined by Δ and the continuous-time state matrix? What supports the proposed direction of optimization pressure under short-horizon credit assignment?
  2. What restrictions, if any, are placed on the learnable gain and bias in MR? Which property of the reparameterization prevents excessively short memory horizons, beyond the non-expansiveness already present in vanilla stable diagonal dynamics?
  3. Why is the entire joint projection normalized rather than the temporal-scale pathway alone? What supports attributing the benefits specifically to Δ stabilization rather than broader changes in input writing, readout, and gating?
  4. How does EF affect the writing of the first observation of a new episode, and under what conditions is its residual dependence on previous episodes sufficiently small? How are convolution and SSM states handled consistently during rollout collection and PPO updates?
  5. Which discretized state update and Mamba version are used? The analysis employs an exact-ZOH input term, while upstream Mamba-2 uses input injection proportional to ΔBx. Please reconcile these formulations and explain how the segmented-reset control relates to the native sequence-ID interface.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.