- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Baleen, a family of state space models that combines an information bottleneck objective with stochastic selection. Two variants use Bernoulli transition gates and Exponential timestep sampling. Token importance scores are derived from average gate values. Experiments cover language tasks, explanation fidelity, and perturbation robustness, with additional analyses of hybrid architectures and information compression.
Critical strengths
- The information bottleneck provides a clear framework for balancing history retention and compression in SSMs. Two concrete mechanisms allow the framework to integrate into existing models.
- The method improves robustness and explanation fidelity across several evaluations while maintaining competitive language task performance. The same-model attribution comparisons in Table 2 provide valuable evidence for the proposed importance scores.
- Hybrid-model experiments, regularization ablations, and compression analysis provide additional evidence about the applicability and behavior of the method.
Critical weaknesses
- The theoretical justification for token importance ranking needs clarification. Section 3.3 ranks tokens by average gate scores. Proposition B.4 establishes coordinate-wise monotonicity and bounds on history-mediated contributions. These results do not generally imply that tokens with higher average scores make larger history-mediated contributions. Unequal channel contributions can reverse this ordering, even under the stated positivity and boundedness assumptions. Table 2 provides empirical support for the scores. Clarifying the additional conditions needed for the claimed ordering and comparing alternative aggregation methods would strengthen the justification.
- Connection between selective retention and robustness. Appendix B.2 provides a theoretical account of perturbation propagation, Tables 3–4 demonstrate robustness gains, and Appendix D.3 reports changes in gating and state compression. These results offer complementary support for the approach. Examining retention coefficients and hidden-state perturbations together in the existing perturbation experiments would help connect the proposed mechanism to the observed prediction stability.
- Performance across context lengths. The language benchmarks and compression analysis provide useful evidence for Baleen. Since the method regulates history retention, understanding how its benefits vary with input length would further clarify its practical scope. A comparison with Mamba2 across several input lengths would make this analysis more complete.
Overall, the results are promising. My main reservations concern the interpretation of the importance scores and the empirical connection between selective retention and robustness, both of which are central to the paper. I currently lean toward weak rejection, while remaining open to revising my assessment based on the author response. The context-length analysis is a secondary concern.
Questions for authors
- What additional assumptions are needed for Proposition B.4 to justify ranking tokens by average gate scores? Could the authors compare uniform averaging with an alternative that accounts for differences in channel contributions?
- Can the existing perturbation experiments provide evidence connecting effective retention coefficients, hidden-state perturbations, and prediction stability?
- Could the authors discuss which context lengths or dependency ranges are likely to benefit most from Baleen, drawing on any available results?