- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TCP-SSM, a replacement for the selective-scan operator in Vision Mamba and EfficientVMamba that is written as a transfer function with explicit poles. Each group of channels shares a small set of real poles and complex-conjugate pole pairs, whose radii and angles are bounded by sigmoid and tanh parameterisations; two scalars predicted from each token rescale all radii and angles, and a low-rank numerator drives the resulting low-order recurrence, which is run in expanded direct form. The paper argues that this preserves pole stability, makes the memory behaviour interpretable through pole radius and angle, and lowers the per-token recurrent cost from to operations per channel. Models are trained with an L1 feature-distillation loss from the original pretrained model plus the task loss, for 600 epochs on Vim-T and 700 epochs on EfficientVMamba-T. TCP-Vim-T1 reduces SSM FLOPs by 40.6% at 76.0% top-1 against 76.1% for Vim-T, and TCP-EVM-B3 reduces them by 44% at 82.0% against 81.8% for EfficientVMamba-B; on Vim-T the paper also reports 41.1 versus 40.6 mIoU on ADE20K and 47.8 versus 45.7 box AP on COCO. The appendix adds matched-compute controls with a smaller Mamba state and end-to-end latency, throughput and memory measurements showing 1.1–1.5× speed-ups.
Critical strengths
Writing the scan as an explicitly factored, bounded, low-order recurrent filter whose radius and angle are modulated per token is a sensible parameterisation: it exposes interpretable quantities such as memory horizon and oscillation frequency, and the cost derivation in App. D is careful. The matched-compute control in App. E.2, which reduces Mamba's state size to the same SSM FLOPs under the same teacher, distillation and schedule, is the right kind of experiment for isolating the operator from the training recipe, and the appendix also reports measured end-to-end latency and memory rather than FLOPs alone.
Critical weaknesses
1. The efficiency claim is not established against the strongest simple alternatives or under a fair training setup.
By the paper's own cost model, reducing Vim's state size from to cuts the per-token cost from 137 to 32, about 116M SSM FLOPs for Vim-T against 295M for TCP-Vim-T1, and VMamba has shown that works with little accuracy loss, yet the matched-compute control stops at . Token pruning and merging methods for Vision Mamba are also missing from the comparison: on Vim-T at 1.2 to 1.3 GFLOPs, DyVM [1], HiddenAlign [2] and R-MeeTo [3] reach 75.1–75.3% and MaMe [4] 76.3% after 3 to 30 epochs of fine-tuning, whereas TCP-Vim-T1 needs by my estimate about 1.3 GFLOPs and 600 epochs for 76.0%. TCP models are trained for 600 to 700 epochs, apparently from teacher-inherited weights and with feature distillation, against baselines trained for 300 epochs, so small gains may come from the schedule rather than the operator. Efficiency is argued through SSM FLOPs, a part of the whole-model cost, which is never reported. latency appears only in the appendix against the authors' own baseline with an unstated scan kernel, and several numbers contradict each other and EfficientVMamba-B configurations in Table 10 whose cost under the paper's model runs opposite to their reported FLOPs.
2. The claimed advantages of the pole formulation, stability, interpretability and novelty, are not supported.
Poles are presented as the key idea from the abstract onwards, but the paper never explains how they relate to a standard selective SSM, for instance that each diagonal entry of Mamba's is already a real pole modulated per token through . Propositions 1 and 2 only show Schur stability of each token's denominator with its poles frozen, while the model runs a direct-form recurrence whose coefficients change at every token; within the paper's own parameter range, with radius 0.99, base angle and the angle scale alternating between 0.8 and 1.2, the product of two consecutive transitions has spectral radius about 1.80, and the gain normalisation cannot prevent this. Stable complex poles parameterised by radius and angle are central to S4D [5], LRU [6] and LinOSS [7], and token-dependent scalar decay to RG-LRU [8] and Mamba-2, yet none is discussed in this context, and no variant without the denominator, with real poles only or without angle modulation is reported.
References
[1] M. Wu, Z. Li, Z. Liang, et al. Dynamic Vision Mamba. arXiv:2504.04787, 2025.
[2] Z. Zhan, Z. Kong, Y. Gong, et al. Exploring Token Pruning in Vision State Space Models. NeurIPS, 2024.
[3] M. Shi, Y. Zhou, et al. Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-training. arXiv:2412.12496, 2024.
[4] J. Park, M. Son, C. Kim. Towards Efficient Vision State Space Models via Token Merging. arXiv:2508.13599, 2025.
[5] A. Gu, A. Gupta, K. Goel, C. Ré. On the Parameterization and Initialization of Diagonal State Space Models. NeurIPS, 2022.
[6] A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, S. De. Resurrecting Recurrent Neural Networks for Long Sequences. ICML, 2023.
[7] T. K. Rusch, D. Rus. Oscillatory State-Space Models. ICLR, 2025.
[8] S. De, S. L. Smith, A. Fernando, et al. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models. arXiv:2402.19427, 2024.
Questions for authors
Q1 Under the same teacher, distillation and schedule, how do conventional SSMs with compare with TCP-SSM in accuracy, whole-model FLOPs and measured latency? How does TCP-SSM compare with HiddenAlign, DyVM, R-MeeTo and MaMe at matched FLOPs and latency?
Q2 Which parameters of the TCP student are initialised from the teacher? Please report an original-operator student with the same initialisation, epochs and distillation, and TCP-SSM trained from scratch for 300 epochs without distillation.
Q3 Can you prove bounded-input bounded-output or exponential stability for the time-varying recurrence that is actually implemented, or report empirical evidence such as the maximum state norm along long scans and at higher resolutions?
Q4 What accuracy do the variants without the denominator, with real poles only and without angle modulation reach, and how does TCP-SSM compare with a complex-diagonal selective SSM such as LRU at the same state budget?
3 comments
[2/2] Response to Reviewer NbS4
2. Initialization and distillation controls. As clarified above, no TCP-specific recurrence parameter is inherited from the teacher. The frozen teacher provides only stop-gradient feature targets in Eq. (15) and is removed at inference. Table 8 provides the original-operator student under the same initialization policy, teacher supervision, training schedule, optimizer, and recurrent-compute budget. Table 4 keeps the TCP architecture and training setting fixed while removing distillation, yielding Top-1. Table 8 then holds teacher supervision, schedule, and recurrent computation fixed while changing the recurrence: TCP-Vim-T1 reaches , compared with for the conventional student. Notably, TCP without distillation already exceeds the distilled compact conventional model, while identical teacher guidance increases the advantage to points. These controls attribute the improvement to the proposed recurrence rather than distillation alone.
3. Stability of the complete token-varying recurrence. Please see our stability response above. In short terms, Proposition 2 guarantees a uniform Schur-stability margin for every token-conditioned denominator. We further show that the bounded recurrent excitation and finite-horizon gain provide an explicit bound on the states and outputs of the complete token-varying scan. We additionally verify this directly on ImageNet using the realized token-conditioned denominators, where the recurrent-state and output norms remain bounded throughout the scans.
4. Pole formulation, component roles, and relation to prior recurrences. A conventional selective SSM produces token-dependent discrete poles through
In Vim, the learned are real and negative, so : token conditioning changes the decay rate of real modes, but does not introduce signed-real responses, complex-conjugate modes, or explicit frequency control. TCP instead parameterizes the full local denominator with mixed real and complex poles,
providing direct control over both memory decay and oscillatory response while preserving the pole constraints of Proposition 2.
The distinction remains even for a complex-diagonal selective SSM. For ,
so the same jointly controls radius and angle:
TCP removes this coupling through independent bounded modulation,
allowing memory duration and oscillatory frequency to adapt independently to the current token.
The mixed pole bank is also order-efficient: a real pole models monotone or sign-alternating decay with one order, whereas a complex-conjugate pair uses two orders for damped oscillation. TCP therefore allocates recurrence order according to the required dynamics rather than using only real modes or unnecessarily representing every mode with a complex pair.
The denominator and numerator serve complementary roles in
determines the recurrent memory and frequency response, while controls how token content excites these modes. Removing removes the autoregressive recurrence itself, rather than an optional TCP component. The low-rank numerator,
simultaneously reduces the dominant excitation cost from to approximately .
LRU is related through its use of stable complex diagonal eigenvalues, but its recurrence uses a single learned diagonal matrix shared across the entire sequence, i.e., static rather than token-conditioned poles. TCP instead generates token-dependent grouped denominators with mixed real/complex modes and independent radius/angle modulation, while retaining selective content-dependent excitation.
5. Table 10 configuration inconsistency. Please see our response above. The inconsistency is limited to a permutation of the TCP-EVM-B1--B5 configuration labels. The trained checkpoints, reported FLOPs and accuracies, Fig. 3, end-to-end runtime and memory measurements, and downstream results were all produced using the correct configurations and remain unchanged.
Follow-Up: Addressing Reviewer Concerns and Request for Reassessment
As the rebuttal discussion period approaches its conclusion, we sincerely thank the reviewer NbS4 for their constructive feedback and thoughtful engagement. We hope that our detailed responses, additional analyses, and new experimental results have adequately addressed the concerns raised. We would greatly appreciate any further feedback on outstanding questions and, if our clarifications have satisfactorily resolved the concerns, kindly invite the reviewers to reconsider their assessments and update their scores accordingly.
[1/2] Response to Reviewer NbS4
We thank the reviewer for the detailed assessment and for recognizing the explicit pole formulation, matched-compute controls, computational analysis, and measured end-to-end efficiency gains. We address the concerns below.
1. Efficiency, matched training, and comparison with token-reduction methods. We first clarify that TCP's accuracy gain is not obtained by comparing a 600/700-epoch TCP model against a shorter-trained conventional baseline. Table 8 isolates the recurrent operator under matched conditions. For Vim-T, the conventional N=8 student and TCP-Vim-T1 use the same pretrained teacher, feature-distillation objective, distilled layers, 600-epoch schedule, cosine scheduler, and optimizer. Likewise, EVM-T (N1,N2)=(8,10) and TCP-EVM-T1 use the same 700-epoch training recipe. Their recurrent computation is also essentially identical:
\begin{array}{lccc} \hline \text{Model} & \\#\text{Param.} & \text{SSM FLOPs} & \text{Top-1} \\\\ \hline \text{Vim-T }(N=8) & 6.7\text{M} & 294.1\text{M} & 75.1 \\\\ \text{TCP-Vim-T1} & \text{7.5M} & \text{295.3M} & \text{76.0} \\\\ \text{EVM-T }(N_1=8,N_2=10) & 6.3\text{M} & 62.9\text{M} & 75.4 \\\\ \text{TCP-EVM-T1} & \text{6.4M} & \text{62.9M} & \text{76.3} \\\\ \hline \end{array}
Thus, TCP yields the same +0.9-point Top-1 gain on both backbones under matched teacher supervision, training schedule, and recurrent computation, isolating the effect of the proposed recurrence.
The initialization is matched as well: both TCP and compact conventional controls initialize all unchanged, shape-compatible backbone parameters from the same pretrained host. TCP's pole bank, modulation heads, grouped denominator, low-rank numerator, and direct path are newly initialized; in the conventional controls, only state-dependent tensors whose dimensions change are newly initialized. TCP therefore receives no initialization advantage.
The reviewer also suggests a substantially smaller conventional state dimension. This is not a matched-compute control for TCP-Vim-T1. We select N=8 because its 294.1M SSM FLOPs nearly exactly match TCP-Vim-T1 at 295.3M.
We also compare with the cited token-reduction methods at similar whole-model compute:
MethodHiddenAlign/ToPDyVMR-MeeToTCP-Vim-T1Efficiency mechanismToken pruning + state alignmentDynamic token/block selectionToken mergingRecurrent-operator replacementTotal FLOPs1.29G1.25G1.28G1.2478GTop-175.175.275.376.0
The initialization regime is important for interpreting this comparison. HiddenAlign, DyVM, and R-MeeTo retain the pretrained recurrent SSM, whereas TCP replaces every Vim SSM with newly initialized pole-structured dynamics. HiddenAlign modifies token pruning and hidden-state alignment; DyVM uses token/block selection; and R-MeeTo merges tokens. Their shorter optimization requirements follow from adapting a pretrained recurrence rather than relearning it.
This distinction makes the external comparison informative: despite learning a newly initialized recurrent operator, TCP-Vim-T1 outperforms all three methods at comparable total FLOPs (see table). These methods are complementary rather than architectural controls for TCP: they reduce token/block computation, whereas TCP restructures the recurrent operator applied to each retained token. The two directions can in principle be combined.
We treat MaMe separately because its FLOP accounting differs. MaMe also retains a pretrained Vim recurrence and applies token merging, but reports a different Vim-T FLOP baseline from our profiling convention and the other cited methods. We could not identify a public implementation associated with the SSM-MaMe result to reconcile the discrepancy. We therefore regard MaMe as a complementary token-merging result rather than a directly FLOP-matched recurrent-operator comparison.
Finally, our efficiency claim concerns inference efficiency: the teacher and feature-distillation loss are removed entirely at inference. SSM FLOPs isolate the operator modified by TCP, while Table 9 measures complete-model latency, throughput, and peak GPU memory under identical hardware, precision, and software settings. TCP-Vim-T1 reduces latency from 35.2 to 24.3 ms at B=128, from 73.8 to 52.3 ms at B=256, and from 163.4 to 124.6 ms at B=512, with higher throughput and lower peak memory throughout. EVM-T and EVM-B show similar end-to-end improvements without a dedicated fused TCP or parallel-prefix scan kernel.