acceptodds
Under review as a conference paper at ICLR 2027

Effects of Input Variability on Long-Term Dependencies in Selective State Space Models

Abstract

State space models (SSMs) have attracted attention for their memory and computational efficiency in modeling long sequences. In particular, Mamba enables modeling of time-varying dynamics through the selective SSM (S6), an input-dependent mechanism, while preserving parallel computation. Despite its empirical success, compared to input-independent SSMs such as S4 and S5, Mamba may underperform on long-sequence tasks that require preserving information regardless of subsequent inputs. Here we analyze how input variability affects long-term dependencies in the S6 recurrence. We show that larger input variance can make it difficult to preserve previous information over long sequences, suggesting that the discretized state matrix should contain values close to at initialization. We further show that under the standard Mamba initialization, increasing the state size shifts the discretized state matrix distribution toward values close to , without increasing the proportion of values close to . These findings lead to a simple initialization approach based on the Gumbel-Softmax trick, which addresses the issues identified in our analysis by modifying the initialization distribution without any architectural changes. We empirically validate the derived initialization properties on synthetic and LRA tasks, showing that the proposed initialization improves performance on tasks that require preserving prior information over long sequences regardless of subsequent inputs, while maintaining comparable performance on context-dependent tasks. These results help explain why input variability can degrade long-term dependencies in selective SSMs.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Mamba, State Space Model, Long-term Dependencies

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4Fc7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper talks about how input variability affects the long term memory in Mamba family models and proposes Gumbel-Softmax based bias initialization

Critical strengths
  • Theorm 1 and 2 have been characterized welland offer useful insight
  • Results are promising as the improvement are significant and the approach is simple which is interesting.
Critical weaknesses

Weakness

  • There is no theoretical justification: although Theorem 1 examines expected log-retention it does not take into account input-dependent state updates and readouts and therefore its relevance to actual memory performance is limited.

  • For the inconsistent initialization analysis, Table 1 uses a value of (\Delta_{\min}=10^{-3}), whereas Path-X uses (10^{-4}). When using the latter value, the analytically estimated probability gain close to one is only 1.12×, which casts doubt on its explanation of the 10.52-point improvement in accuracy.

  • The connection between sensitivity and learning is not clear: while Theorem 2 describes local transition sensitivity, the paper fails to show that maximizing this sensitivity leads to better optimization or to meaningful input selectivity.

  • Limited novelty relative to existing work. The importance of near-one gates for long-term retention and the tradeoff with input sensitivity have already been investigated by Tallec & Ollivier (2018) and Gu et al. (2020). More directly, B2S6 (Yu & Erichson, 2025) studies input-dependent forgetting in Mamba, while Huang et al. investigate Mamba's input selectivity and memory behavior.

  • There is not enough statistical validation since the large variation in the Path-X results (73.34 ± 12.01 for S6) casts doubt on the reliability of the claimed improvements in the absence of paired-seed analysis or confidence intervals.

  • There is limited evidence regarding generalization: the analysis is based on the assumption of Gaussian inputs, even though the initialization statistics neglect the influence of the inputs. The paper does not carry out sufficient verification to confirm that the proposed mechanism remains valid for actual input distributions or after training has taken place.

Questions for authors

Q1. Can the authors isolate input variability in the discretization pathway while keeping the input-update pathway unchanged, to test the causal prediction of Theorem 1?

Q2. Can the authors recompute Table 1's retention and sensitivity statistics using the actual initialization ranges for each LRA task, particularly Path-X, and report the corresponding learned distributions?

Q3. Is there empirical evidence that the proposed near-one and middle-regime probabilities predict memory retention and optimization better than measures such as effective retention timescale or end-to-end input gradients?

Q4. How does the method compare against a tuned negative bias shift or a simple mixture of discretization biases under identical training settings and other related work? If the method is orthogonal how it improves over the related work.

Q5. Can the authors provide paired-seed improvements and confidence intervals for Image, Pathfinder, and Path-X, particularly given the high variance of the Path-X baseline?

Q6. Do the proposed initialization benefits persist for non-Gaussian input distributions and sequences longer than those encountered during training?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 4Fc7 (1/3)

We appreciate your constructive feedback and suggestions.

Q1. Experiments to isolate input variability in the discretization pathway while keeping the input-update pathway unchanged, to test the causal prediction of Theorem 1.

This feedback has helped us further strengthen our contributions. To isolate the effect of input variability on the discretized state matrix, we added Gaussian noise directly to the pre-activation of Aˉt\bar{A}_t at the dummy positions, zt,i=wi⊤xt+bΔ,i+σϵz_{t,i}=\mathbf{w}_i^\top\mathbf{x}_t+b_{\Delta,i}+\sigma\epsilon, while keeping Bˉt\bar{B}_t and CtC_t unchanged. Since the noise is added directly to the pre-activation, its variance cannot be mitigated by learning wi\mathbf{w}_i.

The results are shown in Figure 3 of the revised version. The red dashed line indicates Lλi,j=ln⁡0.01L\lambda_{i,j}=\ln 0.01 computed from Equation (8), set to match the partial retention at bΔ=−3.0b_\Delta=-3.0 for the dummy sequence length of 100 discussed for Figure 2 of the revised version. For both dummy sequence lengths, the settings inside the red dashed line show high accuracy, and those outside show near chance-level accuracy. As the noise standard deviation increases, the red dashed line shifts toward more negative bΔb_\Delta; that is, larger input variance requires a more negative bias. These results suggest that the expected log-retention rate determines whether information is preserved over the dummy sequence, consistent with Theorem 1. The above discussion has been also added to Section 4 of the revised version.

AuthorsOpenReview ↗

Response to Reviewer 4Fc7 (2/3)

Q2. Distribution statistics using the actual initialization ranges for each LRA task, particularly Path-X.

Q3. Empirical evidence that the proposed near-one and middle-regime probabilities predict memory retention and optimization better than measures such as effective retention timescale or end-to-end input gradients.

Thank you for pointing this out. Before presenting the learned distributions, let us first extend our analysis to examine how values close to 1 should be produced.

After training, the discretized state matrix values may become close to 1 by making the input term wi⊤xt\mathbf{w}_i^\top\mathbf{x}_t sufficiently negative, rather than by making the bias term b,i sufficiently negative. In that case, even a small raw input variance can be amplified quadratically by scaling the input weight, which decreases the expected log-retention.

To clarify this point, we added the following remark after Theorem 1 in the revised version:

Remark 1 (Input-weight scaling increases projected input variance). We consider the setting of Theorem 1 with a nonzero input mean. Suppose that xt∼N(μ,Σ)\mathbf{x}_t\sim\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma}), where the input distribution, including μ\boldsymbol{\mu} and Σ\boldsymbol{\Sigma}, is fixed. Write wi=sw^i\mathbf{w}_i=s\hat{\mathbf{w}}_i with a fixed vector w^i\hat{\mathbf{w}}_i and a scale s≥0s\geq0, and assume m=w^i⊤μ<0m=\hat{\mathbf{w}}_i^\top\boldsymbol{\mu}<0 and v0=w^i⊤Σw^i>0v_0=\hat{\mathbf{w}}_i^\top\boldsymbol{\Sigma}\hat{\mathbf{w}}_i>0. Then, the pre-activation zt,i=wi⊤xt+bΔ,iz_{t,i}=\mathbf{w}_i^\top\mathbf{x}_t+b_{\Delta,i} satisfies zt,i∼N(bΔ,i+sm,s2v0).z_{t,i}\sim\mathcal{N}(b_{\Delta,i}+sm, s^2v_0).

This remark indicates that increasing ss shifts the pre-activation mean in the negative direction linearly by s∣m∣s|m|, while increasing its variance quadratically as s2v0s^2v_0. Since v0>0v_0>0, w^i⊤xt\hat{\mathbf{w}}_i^\top\mathbf{x}_t is positive with nonzero probability. For such inputs, softplus(bΔ,i+sw^i⊤xt)\mathrm{softplus}(b_{\Delta,i}+s\hat{\mathbf{w}}_i^\top\mathbf{x}_t) grows linearly in ss. Therefore, λi,j=−αi,jE[softplus(zt,i)]\lambda_{i,j}=-\alpha_{i,j}\mathbb{E}[\mathrm{softplus}(z_{t,i})] diverges to −∞-\infty as s→∞s\to\infty, even though most discretized state matrix values approach 1. In contrast, setting bΔ,ib_{\Delta,i} to a sufficiently negative value shifts the pre-activation mean without increasing its variance, and λi,j→0\lambda_{i,j}\to 0 as bΔ,i→−∞b_{\Delta,i}\to-\infty. Therefore, the discretized state matrix values close to 1 should be obtained through the bias rather than through the input weight.

To examine this in practice, we analyzed the learned distribution of Aˉ\bar{A} on Path-X.

Table R2. Distribution statistics of the discretized state matrix Aˉ\bar{A} after training on Path-X with state size N=32N=32 (mean ±\pm std over five seeds). "WΔxW_\Delta x only" and "bΔb_\Delta only" set the other term to zero.

Model / componentP(Aˉ≥0.9999)P(\bar{A}\ge0.9999)P(Aˉ≥0.999)P(\bar{A}\ge0.999)P(Aˉ≥0.8)P(\bar{A}\ge0.8)P(0.368≤Aˉ≤0.5)P(0.368\le\bar{A}\le0.5)
S6: full(3.9±1.1)×10−4(3.9\pm1.1)\times10^{-4}0.0377±0.00320.0377\pm0.00320.743±0.0160.743\pm0.0160.0404±0.00200.0404\pm0.0020
↳ WΔxW_\Delta x only(1.6±2.0)×10−7(1.6\pm2.0)\times10^{-7}(2.5±3.2)×10−6(2.5\pm3.2)\times10^{-6}(1.0±0.6)×10−4(1.0\pm0.6)\times10^{-4}0.0176±0.00140.0176\pm0.0014
↳ bΔb_\Delta only(1.1±0.3)×10−4(1.1\pm0.3)\times10^{-4}0.0364±0.00260.0364\pm0.00260.744±0.0160.744\pm0.0160.0411±0.00180.0411\pm0.0018
Proposed: full(7.9±1.7)×10−4(7.9\pm1.7)\times10^{-4} (2.0x)0.0432±0.00300.0432\pm0.0030 (1.14x)0.668±0.0200.668\pm0.020 (0.90x)0.0514±0.00460.0514\pm0.0046 (1.27x)
↳ WΔxW_\Delta x only00 (0.0x)(3.5±7.0)×10−7(3.5\pm7.0)\times10^{-7} (0.14x)(1.7±1.0)×10−4(1.7\pm1.0)\times10^{-4} (1.67x)0.0171±0.00040.0171\pm0.0004 (0.97x)
↳ bΔb_\Delta only(3.7±1.1)×10−4(3.7\pm1.1)\times10^{-4} (3.2x)0.0412±0.00300.0412\pm0.0030 (1.13x)0.670±0.0200.670\pm0.020 (0.90x)0.0520±0.00440.0520\pm0.0044 (1.27x)

In S6, some values close to 1 (Aˉ≥0.9999\bar{A}\ge0.9999) are produced by the input term alone, whereas the proposed method has no such values. Moreover, the ratio of values close to 1 produced by the bias alone is 3.2 times higher in the proposed method than in S6.

By Remark 1, values close to 1 produced by the input are affected by the projected input variance, whereas the bias lowers the pre-activation mean without increasing its variance. Thus, Table R2 empirically supports that the proposed initialization mitigates the effect of input variability by producing values close to 1 through the bias rather than through the input term. This may also be related to the larger variance of S6 across runs on Path-X as shown in Table 2. We have added these results, together with those on Pathfinder, to Appendix C.11 of the revised version.

AuthorsOpenReview ↗

Response to Reviewer 4Fc7 (3/3)

The connection between sensitivity and learning is not clear: while Theorem 2 describes local transition sensitivity, the paper fails to show that maximizing this sensitivity leads to better optimization or to meaningful input selectivity.

We agree that Theorem 2 characterizes a local property, rather than the full training dynamics. However, the derivative in Equation (13) appears as a multiplicative factor in the gradients with respect to wi\mathbf{w}_i and bΔ,ib_{\Delta,i} through Aˉ\bar{A}. Therefore, the learning signals from the input become small when Aˉ\bar{A} is close to 0 or 1, whereas the sensitivity is maximized in the middle regime ([0.368,0.5][0.368, 0.5]), and a non-vanishing sensitivity is a necessary condition for learning from inputs.

Regarding the full training dynamics, we experimentally validate this implication on the selective copying task, which requires memorizing tokens at input-dependent positions. In Figure 4 (right) of the revised version, (α,τ)=(0.2,0.8)(\alpha,\tau)=(0.2,0.8) and (0.5,1.0)(0.5,1.0), which have more middle-regime values, learn this task faster than S6. Conversely, decreasing Δmin⁡\Delta_{\min} from 10−310^{-3} to 10−410^{-4}, which reduces P(0.368≤Aˉ≤0.5)P(0.368\le\bar{A}\le0.5) from 0.0600.060 to 0.0400.040, tends to slow learning on this task even though values close to 1 increase in Table 4 and Figure 6 of the revised version.

In the revised version, we have added these results to Table 4 and Figure 6, and discussed them in Section 6 and Appendix C.10.

Q4. Novelty and comparison with simple bias initialization and other related works.

Yu & Erichson (2025) proved that in S6, a single input of sufficiently large magnitude can exponentially reduce the influence of all previous information. Since their analysis considers the same input weights across channels, this forgetting occurs in every channel simultaneously. In practice, however, the official Mamba implementation uses separate input weights for each channel. Therefore, we analyze forgetting in this setting.

Also, Tallec & Ollivier (2018) and Gu et al. (2020) showed that initializing forget-gate biases so that gate values are close to 1 is effective for learning long-term dependencies in gated RNNs. However, the effect of input variability in S6 on preserving previous information over long sequences has not been analyzed. From this perspective, we show that input variability in S6 can degrade long-term dependencies, and characterize the importance of values close to 1 and middle-regime values through Theorems 1 and 2.

In addition, by Remark 1, when the discretized state matrix values are made close to 1 by scaling the input weight, the expected log-retention rate cannot become close to 0 due to the increasing projected input variance. In contrast, a negative bias lowers the pre-activation mean without increasing its variance, and thus the expected log-retention rate can become close to 0. While prior studies have shown the effectiveness of gate bias initialization, this analysis theoretically suggests that values of the discretized state matrix close to 1 should be obtained through the bias rather than through the input weight. We have added this discussion to the related work section in the revised version.

We also compare ours with S6 using smaller Δmin⁡\Delta_{\min}, a simpler way to increase values close to 1, under identical settings in our response above.

Q5. Paired-seed improvements and confidence intervals for Image, Pathfinder, and Path-X.

Our method significantly outperformed standard Mamba on both Image (p=0.032p=0.032) and Pathfinder (p=0.032p=0.032) in separate two-sided exact Mann–Whitney UU tests (five runs per method). The corresponding mean improvements were 4.504.50 and 4.084.08 percentage points, respectively. On Path-X, our method improved accuracy in four of five seeds.

Q6. Benefits persist for non-Gaussian input distributions and sequences longer than those encountered during training.

The Gaussian assumption is used to obtain an explicit variance-based theorem, and the qualitative conclusion does not depend on it.

For any input distribution with E[xt]=0\mathbb{E}[\mathbf{x}_t]=\mathbf{0} and Var(wi⊤xt)=vi>0\mathrm{Var}(\mathbf{w}_i^\top\mathbf{x}_t)=v_i>0, the strict convexity of softplus and Jensen's inequality give E[softplus(bΔ,i+wi⊤xt)]>softplus(bΔ,i).\mathbb{E}[\mathrm{softplus}(b_{\Delta,i}+\mathbf{w}_i^\top\mathbf{x}_t)]>\mathrm{softplus}(b_{\Delta,i}). Consequently, λi,j<−αi,jsoftplus(bΔ,i)\lambda_{i,j}<-\alpha_{i,j}\mathrm{softplus}(b_{\Delta,i}), where the right-hand side is the expected log-retention rate for zero input variance.

Thus, the Gaussian assumption provides strict monotonicity with respect to viv_i, while the qualitative conclusion that nonzero input variability reduces the expected log-retention follows from convexity for any input distribution, as noted after Theorem 1.

This study focuses on the initialization of S6, and extending our analysis to length extrapolation is a valuable direction for future work.

Reviewer_qzasOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how input variability affects long-term memory in Mamba’s selective state-space recurrence. Under Gaussian inputs, it proves that higher projected input variance reduces expected log-retention and identifies a trade-off between retaining information and maintaining input sensitivity. It proposes a Gumbel-Softmax-based bias initialization that balances these properties without changing the architecture, reporting faster learning on synthetic copying tasks and improved Image, Pathfinder, and Path-X accuracy while maintaining comparable performance on other LRA tasks.

Critical strengths
  1. Useful insight into the limits of increasing state size. Equation 2 shows that standard Mamba initialization bounds the number of entries with Aˉ\bar A near 1 independently of state size NN. Figure 1 illustrates this limitation, and Table 3 shows that the proposed initialization at N=32N=32 outperforms standard S6 at N=64N=64, highlighting the importance of initialization.

  2. A simple, practical initialization change. The Gumbel-Softmax bias initialization adds no trainable parameters or inference cost and recovers standard Mamba initialization at (α,τ)=(0,1)(\alpha,\tau)=(0,1). It balances near-1 entries for retention with intermediate values for input sensitivity. Results align with this motivation: Image, Pathfinder, and Path-X accuracy improve, Path-X’s standard deviation decreases from 12.012.0 to 2.62.6, and Text performance remains comparable.

Critical weaknesses

1: Input variability seems not essential to the paper's argument. It enters mainly through Theorem 1, which is essentially Jensen's inequality applied to softplus. The rest of the story stands without it: near-1 values support retention [1,2], mid-range values support selectivity, and the Mamba initialization lacks near-1 values regardless of N. The standard copying task has constant dummy tokens, and so almost no variability, yet it shows clear gains. The title and abstract therefore overstate the role of input variability.

2: The variability mechanism is never tested. The paper does not measure the variance of the Δ pre-activation on LRA tasks. It also does not show that the proposed initialization reduces sensitivity to input variance empirically on LRA, for example by repeating Figure 2 with the new initialization. The LRA gains can be equally consistent with the simpler explanation that the model just has more long-timescale modes.

[1] A. Trockman, H. Harutyunyan, J. Z. Kolter, S. Kumar, S. Bhojanapalli. Mimetic Initialization Helps State Space Models Learn to Recall. arXiv:2410.11135, 2024.
[2] W. L. Tong, A. Lotfi, E. Abbe, K. Vaggelakos, V. Banna, E. Littwin, J. Susskind, C. Pehlevan, E. Malach. On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models. arXiv:2609.16540, 2026.

Questions for authors
  1. Does Theorem 1 capture retention under temporally correlated inputs? Theorem 1 establishes that higher input variance decreases expected log-retention, but this need not imply worse retention on typical sequences or lower expected retention. For example, if the input is constant within a sequence, the median per-sequence decay rate is α·softplus(b), independent of v, and $\mathbb{E}\left[\prod_{t=1}^{T}\bar A_t\right] $ can increase with v. Given that dummy tokens and background pixels can be strongly correlated across time, could the authors clarify when the expected log-retention result supports the broader claim that input variability degrades memory?

  2. Does the proposed initialization reduce sensitivity to input variance in relative terms? The paper argues that near-1 values are insensitive to input variance. This holds only in absolute terms. When b≪−vb \ll -\sqrt{v}, softplus(z)≈ez\mathrm{softplus}(z) \approx e^z, so E[Δ]≈eb+v/2\mathbb{E}[\Delta] \approx e^{b+v/2}. Variance therefore still shortens the effective timescale by a factor of ev/2e^{v/2} regardless of bb. Could the authors report the effective timescale −1/λ-1/\lambda as a function of vv under both the S6 and the proposed initialization? This would show whether the method actually mitigates the effect of variance or simply adds longer-timescale modes.

7 comments
AuthorsOpenReview ↗

Response to Reviewer qzas (1/3)

We appreciate your insightful feedback and suggestions.

W1: Input variability seems not essential to the paper's argument.

Thank you for pointing this out. It allows us to extend our theoretical analysis and gain an additional insight into its initialization.

We agree that the raw input variance in the copying task is small. In this case, for xt=0\mathbf{x}_t=\mathbf{0} as in Table 1, Equation (6) gives λi,j=−αi,jsoftplus(bΔ,i)\lambda_{i,j}=-\alpha_{i,j}\mathrm{softplus}(b_{\Delta,i}), which is determined by the bias alone. The proposed initialization increases the ratio of values that are close to 1 as shown in Table 1, which is consistent with the faster convergence on the copying task in Figure 4 (left) of the revised version.

Moreover, the variance in Theorem 1, vi=wi⊤Σwiv_i=\mathbf{w}_i^\top\boldsymbol{\Sigma}\mathbf{w}_i in Equations (7) and (8), is that of the projected input wi⊤xt\mathbf{w}_i^\top\mathbf{x}_t, not of the raw input. Even when the raw input variance is small, scaling the input weight increases the projected variance quadratically, which limits the ability of the input term to capture long-term dependencies.

To clarify this point, we added the following remark after Theorem 1 in the revised version:

Remark 1 (Input-weight scaling increases projected input variance). We consider the setting of Theorem 1 with a nonzero input mean. Suppose that xt∼N(μ,Σ)\mathbf{x}_t\sim\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma}), where the input distribution, including μ\boldsymbol{\mu} and Σ\boldsymbol{\Sigma}, is fixed. Write wi=sw^i\mathbf{w}_i=s\hat{\mathbf{w}}_i with a fixed vector w^i\hat{\mathbf{w}}_i and a scale s≥0s\geq0, and assume m=w^i⊤μ<0m=\hat{\mathbf{w}}_i^\top\boldsymbol{\mu}<0 and v0=w^i⊤Σw^i>0v_0=\hat{\mathbf{w}}_i^\top\boldsymbol{\Sigma}\hat{\mathbf{w}}_i>0. Then, the pre-activation zt,i=wi⊤xt+bΔ,iz_{t,i}=\mathbf{w}_i^\top\mathbf{x}_t+b_{\Delta,i} satisfies zt,i∼N(bΔ,i+sm,s2v0).z_{t,i}\sim\mathcal{N}(b_{\Delta,i}+sm, s^2v_0).

This remark indicates that increasing ss shifts the pre-activation mean in the negative direction linearly by s∣m∣s|m|, while increasing its variance quadratically as s2v0s^2v_0. Since v0>0v_0>0, w^i⊤xt\hat{\mathbf{w}}_i^\top\mathbf{x}_t is positive with nonzero probability. For such inputs, softplus(bΔ,i+sw^i⊤xt)\mathrm{softplus}(b_{\Delta,i}+s\hat{\mathbf{w}}_i^\top\mathbf{x}_t) grows linearly in ss. Therefore, λi,j=−αi,jE[softplus(zt,i)]\lambda_{i,j}=-\alpha_{i,j}\mathbb{E}[\mathrm{softplus}(z_{t,i})] diverges to −∞-\infty as s→∞s\to\infty, even though most discretized state matrix values approach 1. In contrast, setting bΔ,ib_{\Delta,i} to a sufficiently negative value shifts the pre-activation mean without increasing its variance, and λi,j→0\lambda_{i,j}\to 0 as bΔ,i→−∞b_{\Delta,i}\to-\infty. Thus, the discretized state matrix values close to 1 should be obtained through the bias rather than through the input weight.

AuthorsOpenReview ↗

Response to Reviewer qzas (2/3)

W2: The variability mechanism is never tested.

As discussed above, making Aˉ\bar{A} close to one through wi⊤xt\mathbf{w}_i^\top\mathbf{x}_t increases the projected input variance.

To examine this, we analyzed the learned distribution of Aˉ\bar{A} on Path-X.

Table R2. Distribution statistics of the discretized state matrix Aˉ\bar{A} after training on Path-X with state size N=32N=32 (mean ±\pm std over five seeds). "WΔxW_\Delta x only" and "bΔb_\Delta only" set the other term to zero.

Model / componentP(Aˉ≥0.9999)P(\bar{A}\ge0.9999)P(Aˉ≥0.999)P(\bar{A}\ge0.999)P(Aˉ≥0.8)P(\bar{A}\ge0.8)P(0.368≤Aˉ≤0.5)P(0.368\le\bar{A}\le0.5)
S6: full(3.9±1.1)×10−4(3.9\pm1.1)\times10^{-4}0.0377±0.00320.0377\pm0.00320.743±0.0160.743\pm0.0160.0404±0.00200.0404\pm0.0020
↳ WΔxW_\Delta x only(1.6±2.0)×10−7(1.6\pm2.0)\times10^{-7}(2.5±3.2)×10−6(2.5\pm3.2)\times10^{-6}(1.0±0.6)×10−4(1.0\pm0.6)\times10^{-4}0.0176±0.00140.0176\pm0.0014
↳ bΔb_\Delta only(1.1±0.3)×10−4(1.1\pm0.3)\times10^{-4}0.0364±0.00260.0364\pm0.00260.744±0.0160.744\pm0.0160.0411±0.00180.0411\pm0.0018
Proposed: full(7.9±1.7)×10−4(7.9\pm1.7)\times10^{-4} (2.0x)0.0432±0.00300.0432\pm0.0030 (1.14x)0.668±0.0200.668\pm0.020 (0.90x)0.0514±0.00460.0514\pm0.0046 (1.27x)
↳ WΔxW_\Delta x only00 (0.0x)(3.5±7.0)×10−7(3.5\pm7.0)\times10^{-7} (0.14x)(1.7±1.0)×10−4(1.7\pm1.0)\times10^{-4} (1.67x)0.0171±0.00040.0171\pm0.0004 (0.97x)
↳ bΔb_\Delta only(3.7±1.1)×10−4(3.7\pm1.1)\times10^{-4} (3.2x)0.0412±0.00300.0412\pm0.0030 (1.13x)0.670±0.0200.670\pm0.020 (0.90x)0.0520±0.00440.0520\pm0.0044 (1.27x)

In S6, some values close to 1 (Aˉ≥0.9999\bar{A}\ge0.9999) are produced by the input term alone, whereas the proposed method has no such values. Moreover, the ratio of values close to 1 produced by the bias alone is 3.2 times higher in the proposed method than in S6.

By Remark 1, values close to 1 produced by the input are affected by the projected input variance, whereas the bias lowers the pre-activation mean without increasing its variance. Thus, Table R2 empirically supports that the proposed initialization mitigates the effect of input variability by producing values close to 1 through the bias rather than through the input term. This may also be related to the larger variance of S6 across runs on Path-X as shown in Table 2. We have added these results, together with those on Pathfinder, to Appendix C.11 of the revised version.

AuthorsOpenReview ↗

Response to Reviewer qzas (3/3)

Does Theorem 1 capture retention under temporally correlated inputs?

We agree that temporal correlation can affect the realized retention. However, as stated in Section 4, our analysis does not assume that the inputs are independent across time.

Theorem 1 concerns the expected log-retention rate λi,j=E[log⁡Aˉt,i,j]\lambda_{i,j}=\mathbb{E}[\log\bar{A}_{t,i,j}], which is determined by the marginal distribution of the pre-activation wi⊤xt+bΔ,i\mathbf{w}_i^\top\mathbf{x}_t+b_{\Delta,i} at each time step. Thus, the variance of wi⊤xt\mathbf{w}_i^\top\mathbf{x}_t decreases λi,j\lambda_{i,j} regardless of temporal correlation, and the expected log-retention over LL steps remains Lλi,jL\lambda_{i,j}. Therefore, Theorem 1 remains valid for temporally correlated inputs.

Moreover, as shown in Remark 1, the variance of wi⊤xt\mathbf{w}_i^\top\mathbf{x}_t scales quadratically when wi\mathbf{w}_i is rescaled. Thus, making Aˉt,i,j\bar{A}_{t,i,j} strongly dependent on wi⊤xt\mathbf{w}_i^\top\mathbf{x}_t can amplify even slight raw input variance and thereby decrease the expected log-retention, even for temporally correlated inputs.

Does the proposed initialization reduce sensitivity to input variance in relative terms?

We agree that, for a fixed vv, the variance shortens the effective time scale by a factor of exp⁡(v/2)\exp(v/2) regardless of the bias. However, by Remark 1, making the discretized state matrix value close to one through wi⊤xt\mathbf{w}_i^\top\mathbf{x}_t instead of the bias increases v=s2v0v=s^2v_0 quadratically in ss, whereas a negative bias lowers the pre-activation mean without increasing vv.

Therefore, for the same pre-activation mean, the bias-induced case has a smaller vv and thus a smaller shortening factor exp⁡(v/2)\exp(v/2). Furthermore, making the bias more negative can compensate for this variance-induced shortening.

The rest of the story stands without it: near-1 values support retention [1,2]

Thank you for pointing out these related works [1, 2].

Trockman et al. (2024) [1] propose initializing the discretized state matrix close to 1 and empirically show that it helps Mamba learn copying and associative recall. This is consistent with our Theorem 1, which shows the importance of values close to 1 for preserving previous information. Unlike [1], our Theorem 2 also shows the importance of middle-regime values for input-dependent control. Thus, our bias initialization increases values close to 1 while preserving middle-regime values.

In concurrent work, Tong et al. (2026) [2] theoretically analyze an input-independent decay gate and show that its initial value affects the training dynamics. In contrast, we analyze how input variability in the input-dependent discretized state matrix of S6 affects its ability to preserve previous information over long sequences, and propose a bias initialization based on this analysis.

Tong et al. (2026) also empirically show that the ordering of effective time scales across initializations is largely preserved after training, further supporting the importance of gate initialization.

Please note that, although [1] has been accepted at a workshop, the full paper has not been formally published in a peer-reviewed venue and is available only on arXiv. Similarly, [2] has not been formally published in a peer-reviewed venue and is available only on arXiv. We have added this discussion to the related work section in the revised version.

Reviewer_3JPZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how projected input variance affects expected log-retention under Gaussian marginals. Its analysis of local sensitivity motivates a change to Mamba's bias initialization, whose behavior also depends on state size. Gumbel-Softmax sampling reshapes the distribution to balance slow decay with input sensitivity. Experiments on copying and selective copying are followed by six LRA tasks, with gains over default S6 concentrated on Image, Pathfinder, and Path-X.

Critical strengths
  • The analysis explains how projected input variability affects log-retention and relates it to local sensitivity. Changing only bias initialization helps isolate its role while preserving the architecture and inference cost.
  • Five-seed LRA results improve Image from 77.12 to 81.62. The pathfinding tasks also benefit: Pathfinder rises from 79.56 to 83.64 and Path-X from 73.34 to 83.86, with similar performance retained on Text and Retrieval.
Critical weaknesses
  • Theorem 1 establishes expected log-retention under its assumptions, leaving the connection to expected retention and accuracy to be tested. Theorem 2's local sensitivity result likewise needs validation in the full learning dynamics. Figure 2's noise can affect state writing and optimization, and Table 1's zero-input statistics report tiny near-one probabilities. Measuring retention on actual inputs while intervening only on the step-size branch would help isolate the mechanism.
  • Page 8 says the proposed method converges faster on copying, yet Figure 3 (left) appears to show the orange baseline ahead of the blue proposed curve. Comparisons with simpler bias/step-size adjustments would help assess the value of the chosen distribution. Trockman et al., Mimetic Initialization Helps State Space Models Learn to Recall (2024), is also a relevant missing initialization baseline.
Questions for authors
  • How should the copying claim be reconciled with Figure 3 (left)? Please identify any correction needed and give the convergence criterion with per-seed results.
  • Does the advantage persist against simple bias/step-size adjustments and Mimetic initialization under comparable budgets? This would clarify the value of the proposed retention–sensitivity balance.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 3JPZ (1/2)

We appreciate your valuable feedback and questions.

Theorem 1 establishes expected log-retention under its assumptions, leaving the connection to expected retention and accuracy to be tested.

Figure 2's noise can affect state writing and optimization

These comments have helped us further strengthen our contribution. To isolate the effect of input variability on the discretized state matrix, we added Gaussian noise directly to the pre-activation of Aˉt\bar{A}_t at the dummy positions, zt,i=wi⊤xt+bΔ,i+σϵz_{t,i}=\mathbf{w}_i^\top\mathbf{x}_t+b_{\Delta,i}+\sigma\epsilon, while keeping Bˉt\bar{B}_t and CtC_t unchanged. Since the noise is added directly to the pre-activation, its variance cannot be mitigated by learning wi\mathbf{w}_i.

The results are shown in Figure 3 of the revised version. The red dashed line indicates Lλi,j=ln⁡0.01L\lambda_{i,j}=\ln 0.01 computed from Equation (8), set to match the partial retention at bΔ=−3.0b_\Delta=-3.0 for the dummy sequence length of 100 discussed for Figure 2 of the revised version. For both dummy sequence lengths, the settings inside the red dashed line show high accuracy, and those outside show near chance-level accuracy. As the noise standard deviation increases, the red dashed line shifts toward more negative bΔb_\Delta; that is, larger input variance requires a more negative bias. These results suggest that the expected log-retention rate determines whether information is preserved over the dummy sequence, consistent with Theorem 1. The above discussion has been also added to Section 4 of the revised version.

Comparisons with simpler bias/step-size adjustments would help assess the value of the chosen distribution.

Based on your suggestion, we conducted additional experiments to isolate the effect of changing the step-size initialization. We varied Δmin⁡=10−3\Delta_{\min}=10^{-3} while keeping Δmax⁡=10−1\Delta_{\max}=10^{-1} and all other settings are the same.

We estimated the initial distribution statistics of the discretized state matrix using the same procedure as in Table 1.

Table R1. Initial distribution statistics of the discretized state matrix Aˉ\bar{A} with state size N=32N=32 for different minimum step sizes Δmin⁡\Delta_{\min}.

InitializationP(Aˉ≥0.999)P(\bar{A}\ge0.999)P(Aˉ≥0.8)P(\bar{A}\ge0.8)P(0.368≤Aˉ≤0.5)P(0.368\le\bar{A}\le0.5)
S6 (α=0.0\alpha=0.0, τ=1.0\tau=1.0, Δmin⁡=10−3\Delta_{\min}=10^{-3})0.000002 (1.0x)0.614483 (1.0x)0.059945 (1.0x)
S6 (α=0.0\alpha=0.0, τ=1.0\tau=1.0, Δmin⁡=3×10−4\Delta_{\min}=3\times10^{-4})0.009770 (4885.0x)0.693999 (1.13x)0.047612 (0.79x)
S6 (α=0.0\alpha=0.0, τ=1.0\tau=1.0, Δmin⁡=10−4\Delta_{\min}=10^{-4})0.035749 (17874.5x)0.742382 (1.21x)0.040073 (0.67x)
Proposed (α=0.2\alpha=0.2, τ=0.8\tau=0.8, Δmin⁡=10−3\Delta_{\min}=10^{-3})0.000017 (8.5x)0.556955 (0.91x)0.069450 (1.16x)

As shown in Table R1, decreasing Δmin⁡\Delta_{\min} increases the ratio of values close to 1, but it also decreases the ratio of middle-regime values, P(0.368≤Aˉ≤0.5)P(0.368\le\bar{A}\le0.5).

The copying results are consistent with the discussion in Section 6. Decreasing Δmin⁡\Delta_{\min} increases the ratio of values with Aˉ≥0.999\bar{A}\ge0.999 and leads to faster convergence. More importantly, the selective copying results provide additional empirical support for Theorem 2. As the ratio of middle-regime values decreases, learning on the selective copying task tends to become slower. Conversely, in Figure 4 in the revised version (right), initializations with more middle-regime values learn this task faster. Theorem 2 shows that the middle-regime values enable the model to flexibly adjust to the input. Therefore, decreasing the ratio of middle-regime values makes it more difficult to learn selective copying, which requires the model to preserve information at input-dependent positions. In the revised version, we have added these results to Table 4 and Figure 6, and discussed them in Section 6 and Appendix C.10.

AuthorsOpenReview ↗

Response to Reviewer 3JPZ (2/2)

Theorem 2's local sensitivity result likewise needs validation in the full learning dynamics.

We agree that Theorem 2 characterizes a local property, rather than the full training dynamics. However, the derivative in Equation (13) appears as a multiplicative factor in the gradients with respect to wi\mathbf{w}_i and bΔ,ib_{\Delta,i} through Aˉ\bar{A}. Therefore, the learning signals from the input become small when Aˉ\bar{A} is close to 0 or 1, whereas the sensitivity is maximized in the middle regime ([0.368,0.5][0.368, 0.5]), and a non-vanishing sensitivity is a necessary condition for learning from inputs.

Regarding the full training dynamics, we experimentally validate this implication on the selective copying task, which requires memorizing tokens at input-dependent positions. Please refer to the selective copying results with Table R1 in our response above.

Table 1's zero-input statistics report tiny near-one probabilities.

Although the increase in P(Aˉ≥0.999)P(\bar{A}\ge0.999) appears small in absolute terms, it can be effective for preserving information over long sequences. Since Aˉ\bar{A} is multiplied over the sequence, the information retained after LL steps scales as AˉL\bar{A}^L. For Image and Pathfinder (L=1024L=1024), 0.999L≈0.360.999^L\approx0.36, whereas 0.99L≈3×10−50.99^L\approx3\times10^{-5}. Thus, even small differences among values close to 1 strongly affect the preservation of previous information. Consistent with this, the convergence speed on the copying task follows the order of P(Aˉ≥0.999)P(\bar{A}\ge0.999) across all four settings as shown in Figure 4 in the revised version (left), and the proposed initialization improves accuracy on Image and Pathfinder tasks.

How should the copying claim be reconciled with Figure 3 (left)? Please identify any correction needed and give the convergence criterion with per-seed results.

We examined the per-seed results of Figure 3 (now Figure 4 in the revised version) (left). The proposed initialization consistently converges quickly across all three seeds. In contrast, the S6 baseline converges quickly for only two of the three seeds, while the remaining seed converges substantially more slowly. This variation across seeds leads to the larger standard deviation of the S6 baseline. We therefore revised the manuscript as follows: The model with (α,τ)=(0.2,0.8)(\alpha,\tau)=(0.2,0.8) converges more consistently across random seeds, resulting in faster convergence on average than the baseline.

Does the advantage persist against simple bias/step-size adjustments and Mimetic initialization under comparable budgets?

Regarding the simple bias/step-size adjustments, please refer to our response above.

Thank you for pointing out Trockman et al. (2024). Trockman et al. (2024) propose initializing the discretized state matrix close to 1 and empirically show that it helps Mamba learn copying and associative recall. This is consistent with our Theorem 1, which shows the importance of values close to 1 for preserving previous information. Unlike Trockman et al. (2024), our Theorem 2 also shows the importance of middle-regime values for input-dependent control. Thus, our bias initialization increases values close to 1 while preserving middle-regime values.

Please note that, although this study has been accepted at a workshop, the full paper has not been formally published in a peer-reviewed venue and is available only on arXiv. We have added this discussion to the related work section in the revised version.

Thank you for the clarifications and additional experiments. The targeted noise intervention and step-size initialization comparisons strengthen the empirical support for the proposed mechanism, while the explanation of the results across seeds addresses my concern about Figure 3. I am raising my overall rating.

Thank you for your positive response and for raising your score. Your comments helped us strengthen the contribution of our proposed method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.