acceptodds
Under review as a conference paper at ICLR 2027

Baleen: Self‑Interpretable, Robust SSMs with Stochastic Selective Memory

Abstract

We introduce Baleen, a family of state space models that unifies stochastic selection with information bottleneck to build interpretable and robust long‑context learners. Unlike Mamba’s deterministic gates, Baleen treats selection as a random variable and regularizes it with a closed‑form KL to a sparsity prior: (i) Baleen-B samples Bernoulli state‑transition gates; (ii) Baleen-E samples Exponential time‑intervals. This yields an explicit trade‑off between retention and compression and exposes token‑level selection heatmaps at inference for self‑interpretation. On language benchmarks, Baleen improves average accuracy over Mamba2 by 0.95 points at 370M and 1.38 points at 7B. Baleen is also more robust to localized perturbations and adversarial attacks: under sequence perturbations, it reduces the accuracy drop by over 15%. Finally, Baleen’s self-interpretations achieve higher average fidelity than IG and Grad-CAM across text classification tasks. Overall, despite the robustness–accuracy and interpretability–accuracy trade-offs, Baleen shifts the Pareto frontier outward, delivering better robustness, interpretability, and accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
State-space Models, interpretability, robustness, reliability, trusworthy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PN1nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Baleen, a family of state space models that combines an information bottleneck objective with stochastic selection. Two variants use Bernoulli transition gates and Exponential timestep sampling. Token importance scores are derived from average gate values. Experiments cover language tasks, explanation fidelity, and perturbation robustness, with additional analyses of hybrid architectures and information compression.

Critical strengths
  1. The information bottleneck provides a clear framework for balancing history retention and compression in SSMs. Two concrete mechanisms allow the framework to integrate into existing models.
  2. The method improves robustness and explanation fidelity across several evaluations while maintaining competitive language task performance. The same-model attribution comparisons in Table 2 provide valuable evidence for the proposed importance scores.
  3. Hybrid-model experiments, regularization ablations, and compression analysis provide additional evidence about the applicability and behavior of the method.
Critical weaknesses
  1. The theoretical justification for token importance ranking needs clarification. Section 3.3 ranks tokens by average gate scores. Proposition B.4 establishes coordinate-wise monotonicity and bounds on history-mediated contributions. These results do not generally imply that tokens with higher average scores make larger history-mediated contributions. Unequal channel contributions can reverse this ordering, even under the stated positivity and boundedness assumptions. Table 2 provides empirical support for the scores. Clarifying the additional conditions needed for the claimed ordering and comparing alternative aggregation methods would strengthen the justification.
  2. Connection between selective retention and robustness. Appendix B.2 provides a theoretical account of perturbation propagation, Tables 3–4 demonstrate robustness gains, and Appendix D.3 reports changes in gating and state compression. These results offer complementary support for the approach. Examining retention coefficients and hidden-state perturbations together in the existing perturbation experiments would help connect the proposed mechanism to the observed prediction stability.
  3. Performance across context lengths. The language benchmarks and compression analysis provide useful evidence for Baleen. Since the method regulates history retention, understanding how its benefits vary with input length would further clarify its practical scope. A comparison with Mamba2 across several input lengths would make this analysis more complete.

Overall, the results are promising. My main reservations concern the interpretation of the importance scores and the empirical connection between selective retention and robustness, both of which are central to the paper. I currently lean toward weak rejection, while remaining open to revising my assessment based on the author response. The context-length analysis is a secondary concern.

Questions for authors
  1. What additional assumptions are needed for Proposition B.4 to justify ranking tokens by average gate scores? Could the authors compare uniform averaging with an alternative that accounts for differences in channel contributions?
  2. Can the existing perturbation experiments provide evidence connecting effective retention coefficients, hidden-state perturbations, and prediction stability?
  3. Could the authors discuss which context lengths or dependency ranges are likely to benefit most from Baleen, drawing on any available results?
Reviewer_yv74OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Baleen introduces stochastic selection into state space models, with Bernoulli transition gates in Baleen-B and Exponential timestep sampling in Baleen-E. A KL regularizer is intended to impose an information bottleneck, and expected transition values provide token-level interpretation scores. The paper reports improvements in language-task accuracy, explanation fidelity, and robustness under several tested perturbations.

Critical strengths
  1. The stochastic-selection design is an interesting methodological contribution. I see originality in regularizing selection distributions directly within the SSM recurrence. The two variants address memory selection in different ways: Bernoulli gates control retention, while stochastic timesteps jointly affect retention and writing. This is a useful direction for studying how recurrent models manage their memory.

  2. The design fits existing SSM architectures. It modifies existing transition or timestep variables without adding modules and is demonstrated in both pretraining and fine-tuning. Appendix D.1 also reports improvements on a Mamba3 hybrid backbone, providing some evidence that the idea transfers across configurations.

  3. The empirical results are positive, especially for interpretation. Table 1 reports average accuracy gains of 0.95 percentage points at 370M and 1.38 points at 7B for the best respective variants. Table 2 compares explanation methods on the same Baleen models, helping separate attribution quality from architectural differences. The proposed scores achieve higher average fidelity than the tested gradient-based methods. I regard this as useful evidence for the interpretation score, even if its theoretical meaning remains limited.

Critical weaknesses
  1. The implemented KL penalty is not yet convincingly connected to the claimed information bottleneck. Equation (5) uses a state prior Q(Ht∣Ht−1)Q(H_t \mid H_{t-1}), but Equation (6) constructs the corresponding state distribution through a map containing the input-dependent writing term BtxtB_t x_t. The resulting prior generally depends on the current input, so the input-independent trajectory-prior bound in Appendix A.2 cannot be applied directly.

    Consider a scalar, one-step Baleen-B example with a uniform binary input XX, H0=0H_0=0, and deterministic writing such that H1=XH_1=X. Let the gate posterior and prior both be Bernoulli⁡(1/2)\operatorname{Bernoulli}(1/2). The gate KL is zero, while I(X;H1)=log⁡2I(X;H_1)=\log 2. Thus, the gate penalty alone cannot generally upper-bound the claimed state information.

    Separately, when the previous state is zero, the gate-to-state map is not one-to-one. This invalidates the unconditional claim of KL equality in Equation (6), although an inequality could still hold for appropriately defined distributions.

  2. The application of Proposition B.1 to a trained model needs justification. Appendix B.1 explicitly studies the induced stochastic predictor. My concern is how the result applies when its parameters are learned from the same sample used to compute empirical risk. The proof sketch does not state conditions under which the input-trajectory KL controls this training-sample dependence. A fixed-predictor concentration result would not establish that connection. The authors should clarify the intended scope and supply the relevant assumptions and proof.

  3. Proposition B.4 does not establish the stated ordering property. Its coordinate-wise monotonicity and bounds are valid under the stated assumptions, but they do not imply that a larger average gate produces a larger weighted history contribution. For example, with w=(1,10)w=(1,10), a=(1,0)a=(1,0), and b=(0,0.9)b=(0,0.9), we have mean⁡(a)=0.5>0.45=mean⁡(b)\operatorname{mean}(a)=0.5>0.45=\operatorname{mean}(b), whereas w⊤a=1<9=w⊤bw^\top a=1<9=w^\top b.

    Appendix B.3 appropriately describes the score as a proxy for history-mediated influence. I am comfortable with that limited interpretation, supported by Table 2, but the stronger ordering claim should be corrected. Retention of earlier memory also needs to remain distinct from attribution to the current token.

Questions for authors
  1. How can the implemented gate/timestep KL be related to an input-independent trajectory prior despite the input-dependent state update?

  2. Under what assumptions does Proposition B.1 apply to parameters learned from the training sample?

  3. Can you report vanilla Mamba2 using the same average-gate interpretation score, alongside a matched stochastic-gating baseline without KL regularization? These comparisons would clarify the contribution of SSIB to the reported benefits.

Reviewer_VVbfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Baleen, which incorporates stochastic selection and KL regularization into Mamba-style state-space models. The authors use average expected gate values as token-importance scores and evaluate predictive performance, explanation faithfulness, and robustness through language benchmarks, explanation evaluations, pixel perturbations, and adversarial prompt attacks.

Critical strengths
  1. The method is simple. Both variants modify the existing selection mechanism without requiring a separately trained explanation model.
  2. The robustness results are encouraging. These findings provide empirical support for the method’s potential value.
Critical weaknesses
  1. The theoretical justification and scope of average gate values as attribution scores require further clarification. Average gate values directly characterize the retention of historical states at the current position, whereas current-token writing, subsequent propagation, and the target-output readout also affect the final prediction. In a causal language model, an identical context receives identical raw gate scores when followed by different questions, even when those questions require different evidence. This narrows the applicability of the score as an attribution method, particularly for question- or answer-specific explanations in general-purpose language models. Furthermore, in Proposition B.4, it remains unclear how coordinate-wise monotonicity under fixed nonnegative weights and the contribution bounds in Eq. (62) establish that the average gate score is an “order-preserving proxy,” particularly when different memory components have different weights. Please elaborate on this step, clarify whether additional assumptions are needed, and specify the resulting scope of the interpretation guarantee.

  2. The experimental evidence needs a more direct control and confidence intervals. Table 2 reports Mamba2 only with IG, GradX, and GradCAM, without evaluating an explanation based on its average gate values. The current comparison supports the advantage of Baleen’s Self score over the listed post-hoc methods, but does not distinguish whether the gains come from average-gate scoring itself or from stochastic selection and KL regularization improving the faithfulness of those scores. A Mamba2 average-gate baseline using the same scoring and evaluation procedures would directly help establish the paper’s interpretability contribution. In addition, the experimental protocol and computation of Fidelity AUC require a clearer description. Furthermore, Table 1 shows small accuracy improvements on several tasks and regressions on others. Training variability and evaluation confidence intervals are not reported, making it difficult to assess the stability of the gains.

Questions for authors
  1. Please explain how the coordinate-wise monotonicity and contribution bounds in Proposition B.4 establish the order-preserving conclusion, and whether additional assumptions are required.
  2. Could you provide a Mamba2 baseline using the same average-gate score and fully specify the Fidelity AUC protocol and aggregation across layers? These results would directly affect the assessment of the interpretability contribution of stochastic selection.
  3. Regarding the empirical benefits beyond interpretability, are the main gains in Table 1 stable under training variability and evaluation uncertainty?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.