- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper talks about how input variability affects the long term memory in Mamba family models and proposes Gumbel-Softmax based bias initialization
Critical strengths
- Theorm 1 and 2 have been characterized welland offer useful insight
- Results are promising as the improvement are significant and the approach is simple which is interesting.
Critical weaknesses
Weakness
-
There is no theoretical justification: although Theorem 1 examines expected log-retention it does not take into account input-dependent state updates and readouts and therefore its relevance to actual memory performance is limited.
-
For the inconsistent initialization analysis, Table 1 uses a value of (\Delta_{\min}=10^{-3}), whereas Path-X uses (10^{-4}). When using the latter value, the analytically estimated probability gain close to one is only 1.12×, which casts doubt on its explanation of the 10.52-point improvement in accuracy.
-
The connection between sensitivity and learning is not clear: while Theorem 2 describes local transition sensitivity, the paper fails to show that maximizing this sensitivity leads to better optimization or to meaningful input selectivity.
-
Limited novelty relative to existing work. The importance of near-one gates for long-term retention and the tradeoff with input sensitivity have already been investigated by Tallec & Ollivier (2018) and Gu et al. (2020). More directly, B2S6 (Yu & Erichson, 2025) studies input-dependent forgetting in Mamba, while Huang et al. investigate Mamba's input selectivity and memory behavior.
-
There is not enough statistical validation since the large variation in the Path-X results (73.34 ± 12.01 for S6) casts doubt on the reliability of the claimed improvements in the absence of paired-seed analysis or confidence intervals.
-
There is limited evidence regarding generalization: the analysis is based on the assumption of Gaussian inputs, even though the initialization statistics neglect the influence of the inputs. The paper does not carry out sufficient verification to confirm that the proposed mechanism remains valid for actual input distributions or after training has taken place.
Questions for authors
Q1. Can the authors isolate input variability in the discretization pathway while keeping the input-update pathway unchanged, to test the causal prediction of Theorem 1?
Q2. Can the authors recompute Table 1's retention and sensitivity statistics using the actual initialization ranges for each LRA task, particularly Path-X, and report the corresponding learned distributions?
Q3. Is there empirical evidence that the proposed near-one and middle-regime probabilities predict memory retention and optimization better than measures such as effective retention timescale or end-to-end input gradients?
Q4. How does the method compare against a tuned negative bias shift or a simple mixture of discretization biases under identical training settings and other related work? If the method is orthogonal how it improves over the related work.
Q5. Can the authors provide paired-seed improvements and confidence intervals for Image, Pathfinder, and Path-X, particularly given the high variance of the Path-X baseline?
Q6. Do the proposed initialization benefits persist for non-Gaussian input distributions and sequences longer than those encountered during training?
6 comments
Response to Reviewer 4Fc7 (2/3)
Q2. Distribution statistics using the actual initialization ranges for each LRA task, particularly Path-X.
Q3. Empirical evidence that the proposed near-one and middle-regime probabilities predict memory retention and optimization better than measures such as effective retention timescale or end-to-end input gradients.
Thank you for pointing this out. Before presenting the learned distributions, let us first extend our analysis to examine how values close to 1 should be produced.
After training, the discretized state matrix values may become close to 1 by making the input term sufficiently negative, rather than by making the bias term b,i sufficiently negative. In that case, even a small raw input variance can be amplified quadratically by scaling the input weight, which decreases the expected log-retention.
To clarify this point, we added the following remark after Theorem 1 in the revised version:
Remark 1 (Input-weight scaling increases projected input variance). We consider the setting of Theorem 1 with a nonzero input mean. Suppose that , where the input distribution, including and , is fixed. Write with a fixed vector and a scale , and assume and . Then, the pre-activation satisfies
This remark indicates that increasing shifts the pre-activation mean in the negative direction linearly by , while increasing its variance quadratically as . Since , is positive with nonzero probability. For such inputs, grows linearly in . Therefore, diverges to as , even though most discretized state matrix values approach 1. In contrast, setting to a sufficiently negative value shifts the pre-activation mean without increasing its variance, and as . Therefore, the discretized state matrix values close to 1 should be obtained through the bias rather than through the input weight.
To examine this in practice, we analyzed the learned distribution of on Path-X.
Table R2. Distribution statistics of the discretized state matrix after training on Path-X with state size (mean std over five seeds). " only" and " only" set the other term to zero.
| Model / component | ||||
|---|---|---|---|---|
| S6: full | ||||
| ↳ only | ||||
| ↳ only | ||||
| Proposed: full | (2.0x) | (1.14x) | (0.90x) | (1.27x) |
| ↳ only | (0.0x) | (0.14x) | (1.67x) | (0.97x) |
| ↳ only | (3.2x) | (1.13x) | (0.90x) | (1.27x) |
In S6, some values close to 1 () are produced by the input term alone, whereas the proposed method has no such values. Moreover, the ratio of values close to 1 produced by the bias alone is 3.2 times higher in the proposed method than in S6.
By Remark 1, values close to 1 produced by the input are affected by the projected input variance, whereas the bias lowers the pre-activation mean without increasing its variance. Thus, Table R2 empirically supports that the proposed initialization mitigates the effect of input variability by producing values close to 1 through the bias rather than through the input term. This may also be related to the larger variance of S6 across runs on Path-X as shown in Table 2. We have added these results, together with those on Pathfinder, to Appendix C.11 of the revised version.
Response to Reviewer 4Fc7 (3/3)
The connection between sensitivity and learning is not clear: while Theorem 2 describes local transition sensitivity, the paper fails to show that maximizing this sensitivity leads to better optimization or to meaningful input selectivity.
We agree that Theorem 2 characterizes a local property, rather than the full training dynamics. However, the derivative in Equation (13) appears as a multiplicative factor in the gradients with respect to and through . Therefore, the learning signals from the input become small when is close to 0 or 1, whereas the sensitivity is maximized in the middle regime (), and a non-vanishing sensitivity is a necessary condition for learning from inputs.
Regarding the full training dynamics, we experimentally validate this implication on the selective copying task, which requires memorizing tokens at input-dependent positions. In Figure 4 (right) of the revised version, and , which have more middle-regime values, learn this task faster than S6. Conversely, decreasing from to , which reduces from to , tends to slow learning on this task even though values close to 1 increase in Table 4 and Figure 6 of the revised version.
In the revised version, we have added these results to Table 4 and Figure 6, and discussed them in Section 6 and Appendix C.10.
Q4. Novelty and comparison with simple bias initialization and other related works.
Yu & Erichson (2025) proved that in S6, a single input of sufficiently large magnitude can exponentially reduce the influence of all previous information. Since their analysis considers the same input weights across channels, this forgetting occurs in every channel simultaneously. In practice, however, the official Mamba implementation uses separate input weights for each channel. Therefore, we analyze forgetting in this setting.
Also, Tallec & Ollivier (2018) and Gu et al. (2020) showed that initializing forget-gate biases so that gate values are close to 1 is effective for learning long-term dependencies in gated RNNs. However, the effect of input variability in S6 on preserving previous information over long sequences has not been analyzed. From this perspective, we show that input variability in S6 can degrade long-term dependencies, and characterize the importance of values close to 1 and middle-regime values through Theorems 1 and 2.
In addition, by Remark 1, when the discretized state matrix values are made close to 1 by scaling the input weight, the expected log-retention rate cannot become close to 0 due to the increasing projected input variance. In contrast, a negative bias lowers the pre-activation mean without increasing its variance, and thus the expected log-retention rate can become close to 0. While prior studies have shown the effectiveness of gate bias initialization, this analysis theoretically suggests that values of the discretized state matrix close to 1 should be obtained through the bias rather than through the input weight. We have added this discussion to the related work section in the revised version.
We also compare ours with S6 using smaller , a simpler way to increase values close to 1, under identical settings in our response above.
Q5. Paired-seed improvements and confidence intervals for Image, Pathfinder, and Path-X.
Our method significantly outperformed standard Mamba on both Image () and Pathfinder () in separate two-sided exact Mann–Whitney tests (five runs per method). The corresponding mean improvements were and percentage points, respectively. On Path-X, our method improved accuracy in four of five seeds.
Q6. Benefits persist for non-Gaussian input distributions and sequences longer than those encountered during training.
The Gaussian assumption is used to obtain an explicit variance-based theorem, and the qualitative conclusion does not depend on it.
For any input distribution with and , the strict convexity of softplus and Jensen's inequality give Consequently, , where the right-hand side is the expected log-retention rate for zero input variance.
Thus, the Gaussian assumption provides strict monotonicity with respect to , while the qualitative conclusion that nonzero input variability reduces the expected log-retention follows from convexity for any input distribution, as noted after Theorem 1.
This study focuses on the initialization of S6, and extending our analysis to length extrapolation is a valuable direction for future work.
Response to Reviewer 4Fc7 (1/3)
We appreciate your constructive feedback and suggestions.
This feedback has helped us further strengthen our contributions. To isolate the effect of input variability on the discretized state matrix, we added Gaussian noise directly to the pre-activation of Aˉt at the dummy positions, zt,i=wi⊤xt+bΔ,i+σϵ, while keeping Bˉt and Ct unchanged. Since the noise is added directly to the pre-activation, its variance cannot be mitigated by learning wi.
The results are shown in Figure 3 of the revised version. The red dashed line indicates Lλi,j=ln0.01 computed from Equation (8), set to match the partial retention at bΔ=−3.0 for the dummy sequence length of 100 discussed for Figure 2 of the revised version. For both dummy sequence lengths, the settings inside the red dashed line show high accuracy, and those outside show near chance-level accuracy. As the noise standard deviation increases, the red dashed line shifts toward more negative bΔ; that is, larger input variance requires a more negative bias. These results suggest that the expected log-retention rate determines whether information is preserved over the dummy sequence, consistent with Theorem 1. The above discussion has been also added to Section 4 of the revised version.