- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper modifies the state transition of vision Mamba models in three ways and applies them to ViM-S for images and VideoMamba-S for video. B1 replaces the real diagonal transition with a block-diagonal decay–rotation generator built from 2×2 blocks with data-dependent decay and rotation rate, plus a rank- cross-frequency coupling inside . B2 controls the resulting non-normal dynamics during training through a spectral-norm budget on , a hinge penalty on the largest eigenvalue of the symmetric part, and a per-block reachability-Gramian conditioning penalty. B3 uses bilinear discretisation, motivated as the unique one-step θ-method that maps the imaginary axis bijectively onto the unit circle, in contrast with ZOH's aliasing of rotation rates. Because of the coupling, the recurrence is evaluated token by token rather than with a parallel scan. With a single model of about 26M parameters, the paper reports 83.0% top-1 on ImageNet at 224² and 84.4% at 384², against 80.3% and 80.8% for its baseline; 49.0 versus 44.8 mIoU on ADE20K; 49.4 versus 46.4 AP on COCO with Mask R-CNN 3×; 70.4% versus 67.6% on SSv2; and 97.1% versus 95.3% on Breakfast. Component ablations are an exclusive design on SSv2 and a paired permutation test of the bilinear-by-complex interaction, and the full model runs 29% slower than its baseline.
Critical strengths
The individual theoretical results are correct and combining them for a selective SSM with token-dependent rotation and step size is reasonable. The per-token norm bound does control the time-varying product, unlike frozen-eigenvalue arguments. The SSv2 ablation is also more careful than is typical for backbone papers, with structural null controls in which B2 alone and B2 with B3 reproduce the baseline exactly, and a paired permutation test of the discretisation interaction.
Critical weaknesses
The method is built on a weak baseline and falls clearly behind current vision SSMs
Mamba-B3 is built on ViM-S, an older and weak backbone, and in absolute terms it trails vision SSMs released one to two years ago: VSSD-T reaches 83.7% and M2D-T 84.0% at 4.5 to 4.7 GFLOPs, against 83.0% at 5.8 GFLOPs. The paper sets this gap aside by restricting its claims to a self-defined class of isotropic backbones, and on dense prediction Sec. 5.3 reports the margins over VMamba-T and VSSD-T but does not mention that DAMamba-T, listed in the same Table 3, beats Mamba-B3 on every metric, with 50.3 versus 49.0 mIoU and 50.4 versus 49.4 AP. Only one model of about 26M parameters is reported, with no larger variants and no transfer to a stronger backbone, and throughput and memory are compared only with the paper's own baseline, showing a 29% throughput loss and 21% more memory.
The novelty is narrower than claimed, and the gains are not attributed to the novel part.
The rotation blocks come from Mamba-3, as the paper acknowledges, and bilinear discretisation is the standard choice of S4 and DSS, so the main new element is the low-rank coupling inside . However, data-dependent diagonal-plus-low-rank transitions that couple state channels are exactly what DeltaNet [1], Gated DeltaNet [2], RWKV-7 [3] and DeltaProduct [4] use, and none is discussed, although the paper cites Grazzi et al. (2025), which studies this family; these works also train such transitions with chunkwise-parallel algorithms, which calls into question the claim that the coupled recurrence must run token by token. For B2, the symmetric-part constraint follows Lipschitz RNNs [5], and Gramian-based regularisation during SSM training was proposed in [6]. The experiments then do not isolate the coupling: no model uses the rotation blocks with , the coupling norm is bounded by by construction, the configuration uses a state of 128 per head against 16 for ViM-S and 0.7 more GFLOPs, and the parameter-matched baseline matches neither state size nor FLOPs. The rationale for B3 is also contradicted by the paper's diagnostics, since Table 5 reports mean rotation angles of only 0.19 to 0.29 rad, far from the aliasing regime near that B3 is meant to fix.
References
[1] S. Yang, B. Wang, Y. Zhang, Y. Shen, Y. Kim. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. NeurIPS, 2024.
[2] S. Yang, J. Kautz, A. Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR, 2025.
[3] B. Peng, R. Zhang, D. Goldstein, E. Alcaide, X. Du, et al. RWKV-7 "Goose" with Expressive Dynamic State Evolution. arXiv:2503.14456, 2025.
[4] J. Siems, T. Carstensen, A. Zela, F. Hutter, M. Pontil, R. Grazzi. DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products. NeurIPS, 2025.
[5] N. B. Erichson, O. Azencot, A. Queiruga, L. Hodgkinson, M. W. Mahoney. Lipschitz Recurrent Neural Networks. ICLR, 2021.
[6] P. Schwerdtner, J. Berman, B. Peherstorfer. Hankel Singular Value Regularization for Highly Compressible State Space Models. NeurIPS, 2025.
Questions for authors
(1) Please add the rotation-only transition with on SSv2 and ImageNet, with the same B2 and B3 settings, and report the distribution of over tokens and layers. what fraction ever exceeds ?
(2) What are the state dimension and recurrent FLOPs of the baseline and of each B1 variant? Please add a baseline matched in both state size and FLOPs.
(3) How does the coupled transition differ from the diagonal-plus-low-rank transitions of DeltaNet, Gated DeltaNet, RWKV-7 and DeltaProduct, and can it be computed with their chunkwise-parallel algorithms?
(4) Does B1+B2+B3 improve a strong current backbone such as VSSD-T, M2D-T or DAMamba-T under its original recipe, and what are the throughput and memory of Mamba-B3 compared with these models on the same device?
6 comments
Clarifications About the Contributions and New Results (2/2)
Isolating the coupling (Question 1)
New controls on SSv2 (VideoMamba-S, , bilinear, mean of 5 runs per new cell, top-1):
| No coupling | Low-rank coupling | |
|---|---|---|
| Real diagonal, B2 off | 67.9 | 68.3 |
| Rotation blocks, B2 off | 68.6 | 69.4 |
| Rotation blocks, B2 on | 68.7 | 70.4 |
Coupling adds 0.8 without B2 and 1.7 with the B2 and B3 settings the reviewer asks for, against 0.4 on a real diagonal. The coupling contributes most when it is bounded and conditioned. Rotation alone accounts for 0.7 of the 1.5 that B1 gains without B2, and the introduction will be reworded accordingly. We did not repeat this control on ImageNet-1K due to time constraints.
Coupling budget
The bound applies only when B2 is active; the B1-only and B1+B3 rows train without it ( and in Table 5).
The existing ablation does isolate B2, B3 and their interactions with B1, which the reviewer graciously credits: the null controls reproduce their references exactly, the sub-ablation separates the spectral projection from the conditioning term, and the full model exceeds every pair.
State size and FLOPs (Question 2)
The reviewer is right that the two layouts differ, the value of per-head state of only applies to the variants with B1 active, which use 12 heads, each with and rotation blocks. The baseline and the two variants without B1 use the unmodified ViM-S layer, with a state of 16 per channel and no heads, which is why the baseline matches ViM-S at 26.0M parameters and 5.1 GFLOPs. We will correct the Configuration paragraph to state both layouts. The measurements quoted at in Section 3.2 and in the appendix refer to the B1 state and are unaffected. Matching the total parameter count does not close the gap: the parameter-matched baseline reaches 80.6 against 83.0, at 5.5 against 5.8 GFLOPs.
Mean rotation angle and B3
Thank you for raising this concern. We would like to gently clarify that we do not see a contradiction between the paper's diagnostics and the rationale for B3, for the following reasons:
The caption of Table 5 describes as a location statistic for the rate distribution, and a mean of 0.19 to 0.29 does not bound its tail. The paper's evidence for B3 is the interaction of Table 6, where bilinear adds 0.3 on a real-diagonal state and 1.2 on a complex-block state, and that result stands independently of the mean angle.
Over all layers and tokens of the ZOH model, has a median of 0.22, a 95th percentile of 1.85, a 99th percentile of 3.40, a 99.9th percentile of 5.60, a maximum of 8.4 with 1.5% of observations exceeding , and 6.5% exceeding .
Conclusion
The new results separate rotation from coupling, correct the stated state layout, quantify the efficiency trade-off, and position the method against the delta-rule family; a transfer to a stronger backbone was not run. We appreciate the reviewer's comments and are committed to addressing these points in the revised version. We believe our contributions are valuable to the community researching SSM/Mamba Models and kindly request the reviewer to reconsider their initial score in light of these improvements.
Thanks and Follow-Up
We would like to thank you again for your careful review and constructive suggestions. We hope our responses have addressed your concerns regarding the paper. Please let us know if you have any further questions or if any additional information is required.
Revised manuscript to address comments and concerns has been uploaded as promised.
Clarifications About the Contributions and New Results (1/2)
We thank the reviewer for the detailed review and the references. Below, we address each comment in detail.
"The method is built on a weak baseline and falls clearly behind current vision SSMs"
Scope and choice of backbone
The paper evaluates a state-transition design and does not claim state-of-the-art accuracy. ViM-S was chosen for three reasons:
Within that scope the effect is large for a change confined to the transition: +2.7 top-1 on ImageNet-1K, +4.2 mIoU on ADE20K, +3.0 AP on COCO and +2.8 on SSv2 over the same backbone, at 0.4M added parameters and with a parameter-matched baseline 2.4 points below (Tables 2 to 5).
Absolute standing
We agree that Mamba-B3 trails newer backbones. Section 5.2 reports the gaps to VSSD-T (0.7) and M2D-T (1.0). Section 5.3 lists DAMamba-T in Table 3 but omits it from the text; the text will state that DAMamba-T exceeds Mamba-B3 on both dense tasks (50.3 against 49.0 mIoU, 50.4 against 49.4 AP). The revised paper will state the same to compare transparently against the latest published work. The paper reports one model size, about 26M parameters, matching ViM-S and VideoMamba-S. Larger variants were not trained, and the revised paper will state this as a limitation.
Transfer to a stronger backbone (Question 4)
We agree that transfer to a stronger backbone would strengthen the evidence for general applicability. A controlled comparison requires training both the original backbone and its modified counterpart under the same full ImageNet-1K recipe. Given our computational resources, we cannot complete these experiments before the discussion closes. Transfer also requires architectural adaptation rather than a direct layer replacement, particularly for backbones with different recurrence structures. Meanwhile, we will benchmark the public VSSD-T, M2D-T, and DAMamba-T checkpoints alongside Mamba-B3 on the same device for throughput and memory, under the protocol of Table 1 (batch 128, fp16, 2242 input). These measurements require inference only and will be included in the revision uploaded during the discussion period.
"The novelty is narrower than claimed, and the gains are not attributed to the novel part"
Delta-rule transitions (Question 3)
These works are directly relevant and will be discussed in the updated related works section. Mamba-B3 differs in three respects:
We agree that token-by-token evaluation is not forced by the coupling: the chunkwise algorithm of [1] has been adapted to token-dependent diagonal-plus-low-rank transitions [2], and we withdraw the argument of Section 5.1 that the coupled path must be sequential. We have not implemented a chunkwise kernel for the block-diagonal base. We will cite [3] for the symmetric-part constraint and [4] for Gramian regularization, and correct the Related Work sentence that describes control-theoretic analyses as diagnostics only.
References
[1] S. Yang, B. Wang, Y. Zhang, Y. Shen, Y. Kim. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. NeurIPS, 2024.
[2] S. Yang, J. Kautz, A. Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR, 2025.
[3] N. B. Erichson, O. Azencot, A. Queiruga, L. Hodgkinson, M. W. Mahoney. Lipschitz Recurrent Neural Networks. ICLR, 2021.
[4] P. Schwerdtner, J. Berman, B. Peherstorfer. Hankel Singular Value Regularization for Highly Compressible State Space Models. NeurIPS, 2025.