- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work studies how to enable Mamba-like models to better perform tasks that require state tracking and proposes CARACAL. The authors enrich the way internal memory is updated through more flexible rotation operations. The authors also design a corresponding parallel computation method. In the paper's experiments, the most notable results are observed on synthetic state-tracking tasks, while the effects of this modification on language modeling and runtime efficiency are also evaluated.
Critical strengths
-
The paper presents convincing improvements in state tracking. The authors enrich the state transitions of SSMs through noncommuting rotations, provide a theoretical basis in terms of expressivity, and demonstrate clear improvements on challenging synthetic state-tracking tasks. This provides a valuable approach to enhancing the state-tracking ability of SSMs.
-
The work strikes a good balance between state expressivity and computational efficiency. The authors' design can support efficient parallel computation, and combining improvements in expressivity with a practical computation method is one of the more valuable contributions of this paper.
Critical weaknesses
-
The improvement in state-tracking ability has not yet translated into gains on downstream tasks. The paper demonstrates clear advantages on synthetic state-tracking tasks, but this improvement in ability has not yet resulted in a significant performance advantage on the downstream tasks evaluated.
-
The authors introduce variable rotation axes, a new integration scheme, and some supporting design choices at the same time. A direct comparison between CARACAL and Mamba-3 can show the effect of the overall approach, but relevant ablation experiments are lacking to establish the source of the gains.
Questions for authors
-
Which downstream tasks are most likely to benefit from stronger state-tracking ability? Could the authors further explain the relationship between this improvement in ability and downstream performance?
-
Could the authors add a comparison or ablation of fixed versus input-dependent rotation axes to more directly show how this core design contributes to actual performance?
2 comments
Comment
Thank you for the detailed response and clarifications. The reviewer has no further comments and retains the original rating.
Response to Reviewer oCca
We thank the reviewer for an encouraging and constructive review, and are glad they find the state-tracking improvements convincing and the balance between expressivity and computational efficiency one of the paper's more valuable contributions.
Which downstream tasks benefit from stronger state tracking, and how does this ability relate to downstream performance? (W1, Q1)
Tasks in which a model must follow a state through a sequence of operations. Merrill et al. (ICML 2024) write that "state tracking is at the heart of large language model (LLM) capabilities such as tracking entities in a narrative ..., playing chess under certain notation, or evaluating code". Beck et al. (NeurIPS 2024) likewise note that "state tracking is required to evaluate code or to track entities in a long narrative", and Grazzi et al. (ICLR 2025) list "state-tracking problems such as tracking chess moves or code evaluation". Benchmarks test these abilities directly: entity tracking (Kim and Schuster, ACL 2023) and code execution (CRUXEval; Gu et al., ICML 2024).
Caracal demonstrates strong state-tracking capabilities, which makes it a promising candidate for these tasks. Evaluating it on them is beyond the scope of this work, and we leave it as a future direction; Section 6 of the revised manuscript now says so.
How much do the input-dependent rotation axis and FOH each contribute? (W2, Q2)
The input-dependent rotation axis contributes strongly, and exact FOH performs on par with Mamba-3's rule without its learned parameter. Results and protocol are in the common response and new Appendix D.6.
We would be glad to clarify any point further and to continue the discussion.