acceptodds
Under review as a conference paper at ICLR 2027

Caracal: Noncommuting rotations in state-space models

32 views

Abstract

State-space models (SSMs) combine parallel training with fixed-memory decoding, but diagonal transitions limit their state-tracking capacity: these models cannot solve -complete problems. Non-diagonal transitions expand this capacity in models such as block-diagonal SLiCE and DeltaProduct, but dense block composition and repeated per-token updates increase training cost. The central design problem is to combine non-diagonal state transitions with the efficient parallel training of state-space duality (SSD). We resolve this tension with Caracal, a non-diagonal SSM whose noncommuting rotations compose through quaternions. This structure yields SO3SSD, combining hard state tracking with SSD's efficiency. Caracal provably exceeds the state-tracking capacity of diagonal SSMs and solves the benchmark with a single layer. On finite-state automata, it achieves the highest mean accuracy among evaluated parallel models. We also obtain the lowest measured test perplexities in our language-model comparison, competitive zero-shot accuracy, and leading MAD compression and selective-copying results. Including control preparation, SO3SSD trains faster than every tested baseline at 512K tokens and batch one, outperforming the fastest by even when recurrent baselines use smaller states.

Primary area:
foundation or frontier models, including LLMs
Keywords:
State Space Models, Subquadratic Models, LLMs

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oCcaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies how to enable Mamba-like models to better perform tasks that require state tracking and proposes CARACAL. The authors enrich the way internal memory is updated through more flexible rotation operations. The authors also design a corresponding parallel computation method. In the paper's experiments, the most notable results are observed on synthetic state-tracking tasks, while the effects of this modification on language modeling and runtime efficiency are also evaluated.

Critical strengths
  1. The paper presents convincing improvements in state tracking. The authors enrich the state transitions of SSMs through noncommuting rotations, provide a theoretical basis in terms of expressivity, and demonstrate clear improvements on challenging synthetic state-tracking tasks. This provides a valuable approach to enhancing the state-tracking ability of SSMs.

  2. The work strikes a good balance between state expressivity and computational efficiency. The authors' design can support efficient parallel computation, and combining improvements in expressivity with a practical computation method is one of the more valuable contributions of this paper.

Critical weaknesses
  1. The improvement in state-tracking ability has not yet translated into gains on downstream tasks. The paper demonstrates clear advantages on synthetic state-tracking tasks, but this improvement in ability has not yet resulted in a significant performance advantage on the downstream tasks evaluated.

  2. The authors introduce variable rotation axes, a new integration scheme, and some supporting design choices at the same time. A direct comparison between CARACAL and Mamba-3 can show the effect of the overall approach, but relevant ablation experiments are lacking to establish the source of the gains.

Questions for authors
  1. Which downstream tasks are most likely to benefit from stronger state-tracking ability? Could the authors further explain the relationship between this improvement in ability and downstream performance?

  2. Could the authors add a comparison or ablation of fixed versus input-dependent rotation axes to more directly show how this core design contributes to actual performance?

2 comments
AuthorsOpenReview ↗

Response to Reviewer oCca

We thank the reviewer for an encouraging and constructive review, and are glad they find the state-tracking improvements convincing and the balance between expressivity and computational efficiency one of the paper's more valuable contributions.


Which downstream tasks benefit from stronger state tracking, and how does this ability relate to downstream performance? (W1, Q1)

Tasks in which a model must follow a state through a sequence of operations. Merrill et al. (ICML 2024) write that "state tracking is at the heart of large language model (LLM) capabilities such as tracking entities in a narrative ..., playing chess under certain notation, or evaluating code". Beck et al. (NeurIPS 2024) likewise note that "state tracking is required to evaluate code or to track entities in a long narrative", and Grazzi et al. (ICLR 2025) list "state-tracking problems such as tracking chess moves or code evaluation". Benchmarks test these abilities directly: entity tracking (Kim and Schuster, ACL 2023) and code execution (CRUXEval; Gu et al., ICML 2024).

Caracal demonstrates strong state-tracking capabilities, which makes it a promising candidate for these tasks. Evaluating it on them is beyond the scope of this work, and we leave it as a future direction; Section 6 of the revised manuscript now says so.


How much do the input-dependent rotation axis and FOH each contribute? (W2, Q2)

The input-dependent rotation axis contributes strongly, and exact FOH performs on par with Mamba-3's rule without its learned parameter. Results and protocol are in the common response and new Appendix D.6.

We would be glad to clarify any point further and to continue the discussion.

Reviewer_oCcaOpenReview ↗

Comment

Thank you for the detailed response and clarifications. The reviewer has no further comments and retains the original rating.

Reviewer_cXPxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

CARACAL introduces input-dependent SO(3) rotations into SSMs and uses quaternion prefixes and a coordinate transformation to recover efficient scalar SSD computation. The paper also derives a closed-form first-order-hold update and establishes a state-tracking result under explicit precision assumptions. The paper combines theoretical analysis with experiments on state tracking, associative memory, and kernel performance.

Critical strengths
  1. I find the connection between the model structure and its implementation compelling. Sharing a rotation across state triples keeps the transition product compact as memory width grows. The quaternion representation and SSD reduction turn this observation into a practical algorithm.

  2. The A₅ and FSA results provide convincing evidence of improved state tracking. A single layer passes the A₅ benchmark at every tested length, with multiple successful seeds. On FSA emulation, CARACAL reaches ~97% average accuracy, substantially above the evaluated parallel baselines. The separate length-generalization experiment is also useful.

  3. The implementation and evaluation are documented carefully, including multiple seeds, capacity controls, and timing boundaries.

Critical weaknesses
  1. My main concern is that the two ingredients the paper credits for its gains are never ablated. The main design choices are not isolated. Ablations of input-dependent axes and FOH integration are needed to explain the gains.

  2. The speed advantage is configuration-dependent. CARACAL wins at 512K tokens and batch one but is slower than Mamba-3 at 8K and batch eight. End-to-end timings are missing.

  3. Several claims in the abstract and conclusion read stronger than the results.

3.1 The 1.44x speedup is measured at batch one and 512K tokens. At batch 8 and 8K tokens (Table 12), Mamba-3 is about 2.9x faster with a 64-dim state (3.56 vs. 10.40 ms), and still faster with a 128-dim state.

3.2 The A5 models trained on lengths up to 40 drop to 47.6% at length 640 and 7.4% at 5120, while sLSTM is still at 98.8% at 5120. So “generalize beyond the training lengths” needs a qualifier.

Questions for authors
  1. As stated in W1, how much do input-dependent axes and FOH integration each contribute when the remaining architecture and training setup are held fixed?

  2. What are the full-layer or end-to-end training costs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer cXPx

We thank the reviewer for a careful and constructive review. We are glad they find the link between Caracal's structure and its implementation compelling, the A₅ and FSA evidence convincing, and our documentation careful. Their questions led to the ablations and timings below.


How much do the input-dependent rotation axes and FOH integration each contribute? (W1, Q1)

The input-dependent rotation axis contributes strongly, and exact FOH performs on par with Mamba-3's rule without its learned parameter. Results and protocol are in the common response and new Appendix D.6.


Is the speed advantage over Mamba-3 configuration-dependent, and what are the end-to-end training costs? (W2, Q2)

Yes. For the complete mixer, the ratio of Caracal's training time to Mamba-3's depends on precision and batch size (Mamba-3 with 64 or 128 state coordinates per value):

PrecisionBatchvs. Mamba-3, 64vs. Mamba-3, 128
FP3210.52–0.710.43–0.64
FP3280.69–0.810.63–0.75
FP32160.71–0.800.65–0.74
BF1610.45–0.920.30–0.67
BF1681.14–1.520.85–1.15
BF16161.30–1.530.95–1.15

End to end, in FP32, the precision our language models are trained in, Caracal trains faster than Mamba-3 at both scales, in training tokens per second:

Model45M175–180M
Mamba-3156.1k43.7k
Caracal187.3k58.5k

Setup. All timings use one H200 NVL. The mixer ratios cover head widths 32, 64 and 128 at value width 2,048, with lengths up to 131,072, 32,768 and 8,192 tokens at batch 1, 8 and 16. End-to-end throughput is the median of ten updates after three warm-up updates, with each model's training settings from Appendix D.4. Appendix D.8 now includes both tables (Tables 21 and 22), and Section 5.4 states the result.


Several claims in the abstract and conclusion read stronger than the results. Does the 1.44× speedup hold beyond batch 1 and 512K tokens? (W3, W3.1)

We agree that it described a single configuration, and have removed it from the paper. The abstract and conclusion now report the FP32 end-to-end result, and Section 5.4 and the conclusion say where Mamba-3 leads.

Several claims in the abstract and conclusion read stronger than the results. Does "generalize beyond the training lengths" need a qualifier on A₅? (W3.2)

We agree that the claim needs a qualifier, and have added it to the conclusion (Section 6). Among parallel models, Caracal generalizes best up to length 640 (Table 4). Nonlinear RNNs such as LSTM and sLSTM are stronger still, because a single nonlinear recurrent layer can track any finite group (Merrill et al., ICML 2024, Theorem 5.1). Their recurrence runs one step at a time and cannot be parallelized over the sequence (Beck et al., NeurIPS 2024).

We also retrained Caracal with the released code of Walker et al. (NeurIPS 2025), five seeds with decay and five without (γₙ = 0), all else equal. Without decay, length generalization extends markedly, though fewer seeds fit the training lengths: two of five, against three of five with decay. These two keep 100.0% token accuracy at length 640 and 91.6 and 99.5% at 5,120, against 5.0 to 6.5% with decay. Appendix D.1.2 now reports this result.

We would be glad to clarify any point further and to continue the discussion.

I have read the common response and your reply to my review. Thank you for running the additional experiments. FOH appears comparable to Mamba-3’s rule. The end-to-end timings and the more qualified claims resolve my concerns about the speed comparison. The revised claims also address my concern about length generalization.

The fixed-axis ablation addresses my main concern, but I have a question about it:

(1) In the 45M language-modeling control, the fixed-axis variant reaches 31–36 PPL across six seeds, which is worse than every 45M baseline in Table 3 (about 29–31), including Mamba-3, which also uses commuting rotations. Could you explain this?

(2) In the no-decay setting, only two of five seeds fit the training lengths (three of five with decay).

I am keeping my original rating.

We thank the reviewer for the follow-up and are glad the axis ablation addresses their main concern.

(1) The fixed-axis model is an ablation within Caracal. It removes the input-dependent axis and keeps everything else. All state triples of a Caracal head share one rotation. With the axis fixed, each head is left with a single data-dependent angle per token, which substantially restricts the mixer. The baselines draw their expressivity from mechanisms of their own: Mamba-3 rotates half the state pairs of a head, each at its own data-dependent angle, GDN2 builds on Mamba-2's gated recurrence with a data-dependent decay for each channel and adds delta-rule erasure, KLA scales its writes by a propagated uncertainty, and the Transformer attends over the full context. The control therefore measures what the axis contributes to Caracal, not how commuting rotations perform in general. The case for noncommuting rotations rests on state tracking. Under Merrill et al.'s protocol, fixed-axis models track A₄×ℤ₅ and A₅ in none of three seeds, against two of three with input-dependent axes (Appendix D.6).

(2) On A₅, a task counts as solved when any one of five seeds succeeds. This is the standard KLA reports under (Shaj et al., ICML 2026), and our depth benchmark applies it too (Figure 2). Seeds that miss are normal on this task, which needs its own learning rate and schedule; KLA trains it with its own, while we ran Walker et al.'s released settings unchanged. By this standard, Caracal without decay fits the training lengths and generalizes under Walker et al.'s protocol. The two settings differ by one seed of five, and the seeds that miss behave alike with and without decay. Our conclusion in Appendix D.1.2 rests on the seeds that fit: at length 5,120, 128 times the longest training length, they keep 5.0 to 6.5% token accuracy with decay and 91.6 and 99.5% without, against 99.3 and 99.4% for the best two sLSTM seeds in Table 4.

Reviewer_kNeEOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  1. Method. This paper proposes Caracal, a non-diagonal SSM with input-dependent SO(3) rotations. Unlike Mamba-3's fixed-axis rotations, these transitions need not commute. The authors use quaternion prefix products to transform the keys and queries, reducing the remaining recurrence to scalar SSD. They also derive exact first-order-hold (FOH) input integration and introduce Caracal-Δ, which combines rotational transitions with delta-rule updates.

  2. State tracking and synthetic memory. The authors show that a single Caracal layer can track the group A5, and experimentally solve the A5 benchmark with one layer at all tested lengths (3–20). On four finite-state automata tasks, Caracal achieves 97.14% mean accuracy, compared with 64.12% for Mamba-3 and 76.95% for Gated DeltaProduct (Table 1). On MAD, Caracal-Δ\Delta.achieves 94.0% compression accuracy and 98.9% selective copying accuracy, outperforming additive Caracal and GDN2 on these tasks, although GDN2 performs better on fuzzy recall.

  3. Language modeling. The authors pretrain approximately 45M and 177M parameter models on 1.8B and 10.9B FineWeb-Edu tokens, respectively, using 2K context lengths. At 45M, Caracal achieves 29.04 test perplexity versus 29.52 for Mamba-3, with nearly identical average zero-shot accuracy (38.36% vs. 38.38%). At the larger scale, Caracal achieves 18.09 perplexity, compared with 18.86 for GDN2 and 18.80 for KLA, although Mamba-3 perplexity is not reported. The larger-scale zero-shot accuracy is competitive but does not improve over Mamba-3.

  4. Systems. The authors implement SO3SSD and benchmark the additive Caracal recurrence against optimized recurrent and attention operators on an H200. At batch one and 512K tokens, SO3SSD achieves a 1.44x forward-plus-backward speedup over the fastest tested baseline, Mamba-3, despite using a larger recurrent state. The advantage depends on the configuration: Mamba-3 is faster at batch eight and 8K tokens. The reported kernel benchmarks do not include Caracal-Δ\Delta.

Critical strengths
  1. An efficient construction for noncommuting SSMs. The main contribution, in my view, is a practical way to introduce noncommuting transitions into a Mamba-style SSM while retaining scalar SSD. By computing ordered rotation products through quaternions and absorbing them into the keys and queries, the authors preserve the efficient matrix-multiplication structure of SSD. While the underlying group-tracking capability has been established in prior work, I find this construction and its GPU implementation useful contributions.

  2. Strong state-tracking results and an efficient implementation. Caracal substantially improves over Mamba-3 and other parallel recurrent models on finite-state automata, providing convincing empirical evidence for the benefit of noncommuting transitions on these tasks. The kernel benchmarks also demonstrate that these dynamics can be implemented efficiently, particularly at long sequence lengths and small batch sizes, with rotation and integration preparation included in the measurements. I find these contributions valuable even though the broader benefits for language modeling remain less clear.

Critical weaknesses
  1. The empirical benefit of exact FOH integration is not established.

    Caracal introduces both noncommuting rotations and exact first-order-hold (FOH) integration, but the experiments do not isolate their contributions. In particular:

    • State tracking: The expressivity construction in Theorem 4 suppresses input writes, so it does not establish any benefit from FOH integration.
    • Language modeling: Table 3 shows competitive perplexity, but does not establish whether exact FOH integration improves performance over Mamba-3's exponential-trapezoidal rule.

    I would like to see a controlled comparison of the two integration rules while retaining Caracal's input-dependent rotations. This would clarify whether exact integration provides an empirical benefit beyond the mathematical derivation.

  2. The contribution of rotations to Caracal-Δ\Delta need to be better studied.

    The MAD experiments show that adding delta-rule updates to Caracal improves compression (90.7% to 94.0%) and selective copying (78.6% to 98.9%). The authors also provide useful parameter-count controls in Table 7. However, these experiments do not establish whether noncommuting rotations provide an additional benefit over the delta rule itself.

    A useful ablation would retain Caracal-Δ\Delta's delta-rule update and surrounding architecture, but disable the rotations. This would help establish whether combining the two mechanisms is beneficial, particularly given that delta-rule models already perform well on associative memory tasks.

  3. The practical benefits for language modeling and retrieval remain unclear.

    The language-modeling results are competitive, but the evaluation does not yet establish where the additional dynamics are useful.

    • Language modeling: At 45M parameters, Caracal improves perplexity over Mamba-3 (29.04 vs. 29.52), but downstream accuracy is essentially unchanged. At 175–180M parameters, Caracal achieves 18.09 perplexity, but the corresponding Mamba-3 result is missing from Table 3. Downstream accuracy is slightly lower than Mamba-3 (42.80% vs. 43.48%).
    • Retrieval: The paper does not evaluate retrieval in pretrained language models. I would like to see NIAH/RULER evaluations at the trained 2K context length, comparing Caracal against Mamba-3 and the other LM baselines. This would help establish whether the improved state-tracking capability translates to better use of contextual information.
    • Caracal-Δ\Delta: The delta-rule variant is evaluated only on MAD. It would be useful to understand whether its synthetic memory improvements translate to language modeling or retrieval, and what additional computational cost it incurs.

    I do not regard the relatively small scale of the LM experiments as a major weakness, since the noncommuting SSD construction is itself a useful contribution. However, broader retrieval evaluations and experiments at longer, appropriately trained context lengths would help establish the practical benefits of these dynamics.

  4. The efficiency tradeoff from sharing rotations is not quantified.

    Caracal applies the same SO(3) rotation to all three-dimensional state blocks within a head. This allows the quaternion prefix scan to be shared across blocks, reducing the cost of preparing the SSD operands. However, it also restricts the state dynamics, since all blocks within a head undergo the same rotation.

    • Computational cost: The paper explains the benefit of sharing rotations analytically, but does not quantify the overhead of computing additional independent rotation prefixes. These scans can, in principle, be parallelized across blocks, although the additional computation and memory traffic may be significant.
    • Design tradeoff: It would be useful to compare the current shared-rotation design against a configuration with a small number of independent rotation groups, keeping the total state size fixed. Reporting both runtime and performance on state-tracking or memory tasks would help establish whether the restriction is necessary for efficiency or represents a meaningful expressivity tradeoff.
Questions for authors
  1. Contribution of exact FOH integration.

    Could the authors compare exact FOH integration against Mamba-3's exponential-trapezoidal rule, while retaining Caracal's input-dependent rotations?

    I would be interested in a controlled comparison using the existing 45M language-modeling setup, or a representative MAD task, keeping the architecture, training data, and optimization budget unchanged. This would help establish whether exact integration improves empirical performance over the simpler discretization.

  2. Contribution of rotations to Caracal-Δ\Delta.

    Could the authors add a Caracal-Δ\Delta ablation that disables rotations while retaining the delta-rule update, state size, and surrounding architecture? The integration coefficients should be recomputed consistently with the modified transition. Comparing compression, selective copying, and fuzzy recall against full Caracal-Δ\Delta would help determine whether rotations provide an additional benefit beyond the delta rule.

    A fixed-axis rotation control would further help isolate the contribution of noncommutativity, rather than rotations in general.

    Could the authors also report the training and decoding cost of Caracal-Δ\Delta relative to additive Caracal and GDN2? The existing SO3SSD benchmarks cover only the additive variant.

  3. Language modeling and retrieval performance.

    • Missing perplexity: Could the authors report the Mamba-3 test perplexity at 175–180M parameters in Table 3, using the same evaluation procedure as the other models? If this result is unavailable, could they clarify why?
    • Retrieval: Could the authors evaluate the existing 2K-pretrained Caracal and Mamba-3 models on NIAH/RULER retrieval tasks within their trained context length? Reporting accuracy across evidence positions would help establish whether Caracal's stronger state-tracking capability translates to improved retrieval in language models.

    As a longer-term extension, it would also be interesting to compare models trained or adapted to longer contexts (e.g., 32K), measuring held-out long-document perplexity, loss by token position, and retrieval. This would separate limitations of the architecture from those of evaluating beyond its trained context length.

  4. Tradeoff from sharing rotations across state triples.

    Could the authors quantify the efficiency benefit of sharing one SO(3) rotation across all state triples within a head?

    • Computational cost: How much of the existing runtime is spent on rotation-prefix scans and FOH-coefficient preparation relative to the SSD contractions? How would these costs change with multiple independent rotation groups per head, including the additional control preparation?
    • Quality–efficiency tradeoff: Could the authors compare the current design against a configuration with a small number of independent rotation groups, keeping the total state size fixed? Reporting runtime alongside performance on FSA or MAD tasks would help establish whether the shared-rotation restriction is an important practical tradeoff.
2 comments
AuthorsOpenReview ↗

Response to Reviewer kNeE (1/2)

We thank the reviewer for a thorough and insightful review. We are glad they find the quaternion construction that retains scalar SSD, its GPU implementation and the state-tracking results valuable contributions.


Does exact FOH integration improve over Mamba-3's exponential-trapezoidal rule when Caracal's input-dependent rotations are retained? (W1, Q1)

The input-dependent rotation axis contributes strongly, and exact FOH performs on par with Mamba-3's rule without its learned parameter. Results and protocol are in the common response and new Appendix D.6.

Theorem 4's construction does not involve FOH; Section 4.4 now says so.


Do rotations, and noncommuting ones in particular, provide a benefit beyond the delta rule in Caracal-Δ? (W2, Q2)

Rotations add to the delta rule, and noncommuting rotations add to commuting ones:

  • Rotations against no rotation. With rotations, Caracal-Δ leads every model on MAD compression and selective copying (Table 2).
  • Noncommuting against commuting. A fixed axis makes the rotations commute. On MAD, commuting rotations recover most of the gain in Caracal-Δ. Noncommutativity is decisive in state tracking: commuting rotations cannot faithfully represent a non-abelian group, and in base Caracal a fixed axis fails to learn A₄×ℤ₅ and A₅ with one layer and hurts perplexity and FSA accuracy (common response).

The table gives MAD test accuracy (%, five seeds):

ModelCompressionFuzzy recallSelective copy
Noncommuting94.52±0.4660.74±15.7199.13±0.64
Commuting (fixed axis)93.93±0.6965.40±17.6897.56±1.78
No rotations77.59±3.7567.16±8.9558.96±3.90

Setup. Without rotations, the rotation angle is zero and the FOH coefficients are recomputed by the same exact formula, so they match the rotation-free transition. The fixed axis is as in the common response, and everything else is unchanged. New Appendix D.3 reports the table, with the base Caracal rows.

What are the training and decoding costs of Caracal-Δ relative to base Caracal and GDN2? (Q2)

As expected, Caracal-Δ costs more than base Caracal to train, and about the same to decode. It trains faster than GDN2 in FP32, the precision our language models are trained in, and decodes faster in every setting. The table gives the ratio of Caracal-Δ's time to each model's, by precision and head width:

PrecisionHead widthTraining vs. CaracalTraining vs. GDN2Decoding vs. CaracalDecoding vs. GDN2
FP32321.36–1.530.63–0.781.01–1.040.14–0.38
FP32641.22–1.330.50–0.651.00–1.060.14–0.34
FP321281.15–1.250.46–0.610.94–1.050.14–0.31
BF16321.93–2.171.21–1.621.03–1.080.13–0.35
BF16641.67–2.010.92–1.200.99–1.120.14–0.31
BF161281.47–1.860.84–1.100.88–1.170.17–0.33

How much runtime do rotation-prefix scans and FOH coefficients take relative to the SSD contractions, and with independent rotation groups? (W4, Q4)

A small share; the SSD contractions dominate. With groups, the prefix and FOH scans grow but stay well below the contractions. A head with G groups splits its triples into G equal sets, each with its own axis and angle, and shares its decay, gain, keys, queries and values. The added control projection changes dense-layer time by at most 0.5 ms. Because the groups share the decay, the SSD contractions keep their shape and apply G prefixes to keys and queries (Proposition 3). The table gives training time by stage (ms), with the prefix and FOH time at two and four groups:

Head widthPrefixes, FOHContractionsRecurrenceDenseOtherTotal2 groups4 groups
321.4111.351.246.570.2420.812.554.99
640.766.940.986.110.2415.031.322.54
1280.607.000.995.920.4614.971.172.24

End to end, without a dedicated kernel, each group runs as a separate head with 1/G of the state, which repeats the value-side contraction (training time and additional peak memory relative to one group, over shapes):

PrecisionGroupsTime, medianTime, rangeMemory, medianMemory, range
FP3221.241.20–1.321.121.01–1.20
FP3242.191.73–2.801.581.04–1.68
BF1621.501.41–1.611.171.13–1.25
BF1644.052.39–4.961.571.43–1.69

Setup. All timings use one H200 NVL and the complete mixer at value width 2,048. Training covers lengths up to 131,072, 32,768 and 8,192 tokens at batch 1, 8 and 16, and decoding one token at batch 1 to 128; four groups ran out of memory at two FP32 shapes, batch 8 at 32,768 tokens. The stage profile is batch 8 at 8,192 tokens in BF16, with 96 state coordinates per value. Appendix D.8 now includes the three tables.

AuthorsOpenReview ↗

Response to Reviewer kNeE (2/2)

What is Mamba-3's test perplexity at 175–180M? (W3, Q3)

Mamba-3's test perplexity at 175–180M is 18.30, against Caracal's 18.09, with the same evaluation procedure as the other models. We apologize for omitting it; Table 3 now includes it.

Caracal improves perplexity over Mamba-3, but downstream accuracy is essentially unchanged. Where are the additional dynamics useful? (W3)

Caracal has the lowest test perplexity at 175–180M, and Caracal-Δ and Caracal the two lowest at 45M (Table 3). The eight zero-shot tasks probe commonsense and reading comprehension rather than state tracking, which is what Caracal's added dynamics target; we discuss where we expect the gains in our reply to Reviewer oCca.


Does Caracal's stronger state tracking translate to better retrieval than Mamba-3 on RULER within the trained context? (W3, Q3)

Yes. Caracal retrieves better than Mamba-3 at both scales and both lengths, and Caracal-Δ retrieves best of the three throughout (RULER average, %, eleven synthetic tasks):

Model45M, 1,02445M, 2,048175–180M, 1,024175–180M, 2,048
Mamba-33.53±0.541.40±0.3021.938.28
Caracal12.49±1.995.27±0.8522.539.44
Caracal-Δ19.73±0.9214.16±0.9822.9117.13

How does NIAH accuracy vary across evidence positions? (Q3)

Caracal retrieves from further back in the context than Mamba-3 at 45M, and Caracal-Δ retrieves at every depth at both scales (needle accuracy, %, at 2,048 tokens, by depth in tenths of the input):

Model0–1010–2020–3030–4040–5050–6060–7070–8080–9090–100
45M
Mamba-30.000.000.000.000.000.000.001.515.6914.75
Caracal0.000.000.000.120.501.481.8810.3225.5345.93
Caracal-Δ11.0212.3113.9313.4516.0714.1019.8227.6337.9049.92
175–180M
Mamba-30.000.000.291.092.692.666.9825.4834.7853.46
Caracal0.390.000.000.360.003.8516.2828.7141.1451.61
Caracal-Δ15.7517.8517.0015.6418.5615.0924.5829.3539.1348.85

Setup. We score the final checkpoints on RULER's eleven synthetic tasks: eight needle-in-a-haystack variants, variable tracking, and common- and frequent-word extraction, with 500 samples per task and length from RULER's own generators. The two question-answering tasks are omitted because their prompts exceed 1,024 tokens. New Appendix D.5 gives both tables for all language-model baselines, with the essay test.


Do Caracal-Δ's synthetic memory improvements translate to language modeling or retrieval? (W3)

Yes. Caracal-Δ has the lowest 45M test perplexity of all models and the highest RULER average at 2,048 tokens of any recurrent model at both scales (Table 12). Its test perplexity is 28.95 ± 0.35 at 45M, and 18.32 at 175–180M, on par with Mamba-3's 18.30. Table 3 and Appendix D.4 now include it.

How do models trained or adapted to longer contexts, such as 32K, compare? (W3, Q3)

We agree that this comparison would be valuable and leave it to future work.


How does the shared-rotation design compare with a small number of independent rotation groups at fixed total state size? (W4, Q4)

Sharing one rotation is an efficiency choice: it matches rotation groups on FSA and costs selective-copying accuracy on MAD. On FSA, Theorem 4's construction needs only one rotation per head (test accuracy, %, five seeds):

ModelCycle nav.Even pairsMod. arith.ParityAverage
Caracal99.99±0.0299.90±0.2097.62±1.76100.00±0.0199.38±0.47
2 rotation groups98.98±2.29100.00±0.0095.57±6.4599.97±0.0798.63±1.49
4 rotation groups99.99±0.0299.56±0.6297.14±4.5899.97±0.0699.16±1.15

On MAD, each result is the mean of runs on two GPU types (test accuracy, %, five seeds):

ModelCompressionFuzzy recallSelective copy
Caracal91.91±0.7446.57±8.4878.83±8.62
2 rotation groups91.79±0.6246.93±3.3592.08±5.03
4 rotation groups92.42±0.5846.88±9.1399.03±0.44

Setup. Only the number of groups changes; groups add 1.0 to 4.8% parameters through the control projection. The Caracal reference is retrained alongside, with the protocols of Table 1 (FSA) and Appendix D.2 (MAD). New Appendix D.7 defines rotation groups and gives both tables.

We would be glad to clarify any point further and to continue the discussion.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common response

We sincerely thank the reviewers for their careful and constructive reviews. We are encouraged that they (kNeE, cXPx, oCca) find Caracal's combination of noncommuting rotations with efficient SSD computation valuable and its state-tracking results convincing. Here we address common concerns. Specific concerns are addressed in the individual responses.


How much do the input-dependent rotation axes and exact first-order-hold (FOH) integration each contribute, against a fixed axis and Mamba-3's exponential-trapezoidal rule? (kNeE W1, Q1; cXPx W1, Q1; oCca W2, Q2)

The input-dependent rotation axis contributes strongly to Caracal's improvements:

  1. A fixed axis hurts perplexity in every seed.
  2. It lowers accuracy on FSA and MAD selective copying.
  3. With one layer, it fails to learn both non-abelian groups, A₄×ℤ₅ and A₅.

Exact FOH performs on par with Mamba-3's rule without its extra learned parameter. New Appendix D.6 reports both controls, each trained from scratch with the same recipe and seeds. We give details below.

Input-dependent rotation axis. In the control, each head's rotation axis is a learned constant, not a function of the input; the rotation angle stays input-dependent, and the rest is unchanged.

In 45M language modeling, the fixed axis hurts test perplexity in every seed:

Variants0s1s2s3s4s5Paired diff.
Caracal28.9329.0129.0829.0429.1428.95
Fixed axis36.4431.5135.3034.3734.1633.70+5.22±1.67

On MAD, it lowers test accuracy on selective copying (%, five seeds):

VariantCompressionFuzzy recallSelective copy
Caracal91.12±0.8447.31±8.1679.93±10.05
Fixed axis91.27±0.9150.74±5.3866.75±6.93

On FSA, it lowers test accuracy (%, five seeds) on modular arithmetic and parity:

VariantCycle nav.Even pairsMod. arith.ParityAverage
Caracal98.35±3.68100.00±0.0190.52±17.18100.00±0.0197.22±5.21
Fixed axis100.00±0.0199.81±0.2874.24±12.5583.70±17.7689.44±3.47

On group word problems, it fails to learn both non-abelian groups (seeds of three that learn each with one layer; Merrill et al.'s protocol):

Axesℤ₆₀ (abelian)A₄×ℤ₅A₅
Input-dependent322
Fixed axis300

Exact FOH integration. Mamba-3's learned rule replaces FOH's coefficients; the rest is unchanged. In 45M language modeling, it matches FOH in five of six seeds:

Variants0s1s2s3s4s5Paired diff.
Caracal28.9329.0129.0829.0429.1428.95
Mamba-3's rule29.1229.0929.0331.8828.9828.86+0.47±1.17

In seed s3, Mamba-3's rule had gradient-norm spikes and ended at 31.88. On MAD, FOH leads all three tasks within one standard deviation: 91.12, 47.31 and 79.93% against 90.48, 44.46 and 78.27%.

Rotations about a fixed axis commute, as in Mamba-3, so they cannot faithfully represent a non-abelian group such as A₅. Noncommuting rotations therefore contribute strongly to Caracal's gains. Exact FOH matches Mamba-3's rule without its extra learned parameter.


Revisions. We uploaded a revised PDF, with changes marked in blue and every new experiment in the appendix:

  1. Section 4.4 states that Theorem 4's construction does not involve FOH (kNeE W1).
  2. Section 6 names downstream state-tracking tasks as future work (oCca W1, Q1).
  3. Appendix D.1.2 adds A₅ length generalization without decay (cXPx W3.2).
  4. New Appendix D.3: Caracal-Δ without rotations and with a fixed axis (Table 8; kNeE W2, Q2).
  5. Table 3 adds Mamba-3 at 175–180M and Caracal-Δ at both scales; Appendix D.4 gives Caracal-Δ's configuration (kNeE W3, Q3).
  6. New Appendix D.5: RULER and needle retrieval within the trained context (Tables 12 and 13; kNeE W3, Q3).
  7. New Appendix D.6: the integration-rule and fixed-axis controls above (Tables 14 to 16; kNeE W1, Q1; cXPx W1, Q1; oCca W2, Q2).
  8. New Appendix D.7: rotation groups on FSA and MAD (Tables 17 and 18; kNeE W4, Q4).
  9. Appendix D.8 adds complete-mixer and end-to-end training, decoding, stage timings and Caracal-Δ's cost (Tables 21 to 26; cXPx W2, Q2; kNeE Q2, W4, Q4).
  10. The 1.44× result is removed from the abstract, introduction, Section 5.4 and conclusion, and the conclusion limits length generalization on A₅ to fourfold (cXPx W3).

The submission's Appendices D.3 and D.4 are now D.4 and D.8, and its Table 12 is Table 20.

Area_Chair_6izMOpenReview ↗

Reminder to reviwers to read author's responses

Dear Reviewers,

Thank you for your efforts in providing thorough reviews.

As the discussion period is underway, I would like to kindly remind you to read the authors' responses to your reviews and provide follow-up comments where appropriate or leave a comment. In particular, please consider whether the responses address your questions and concerns, and update your assessment if the clarifications or additional information affect your initial evaluation.

Your engagement in this discussion is important for ensuring a fair and thorough review process.

Thank you again for your time and contributions!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.