Effects of Input Variability on Long-Term Dependencies in Selective State Space Models

State space models (SSMs) have attracted attention for their memory and computational efficiency in modeling long sequences. In particular, Mamba enables modeling of time-varying dynamics through the selective SSM (S6), an input-dependent mechanism, while preserving parallel computation. Despite its empirical success, compared to input-independent SSMs such as S4 and S5, Mamba may underperform on long-sequence tasks that require preserving information regardless of subsequent inputs. Here we analyze how input variability affects long-term dependencies in the S6 recurrence. We show that larger input variance can make it difficult to preserve previous information over long sequences, suggesting that the discretized state matrix should contain values close to $1$ at initialization. We further show that under the standard Mamba initialization, increasing the state size shifts the discretized state matrix distribution toward values close to $0$, without increasing the proportion of values close to $1$. These findings lead to a simple initialization approach based on the Gumbel-Softmax trick, which addresses the issues identified in our analysis by modifying the initialization distribution without any architectural changes. We empirically validate the derived initialization properties on synthetic and LRA tasks, showing that the proposed initialization improves performance on tasks that require preserving prior information over long sequences regardless of subsequent inputs, while maintaining comparable performance on context-dependent tasks. These results help explain why input variability can degrade long-term dependencies in selective SSMs.