acceptodds
Withdrawn from ICLR 2027

TCP-SSM: Efficient Vision State Space Models with Token-Conditioned Poles

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

State Space Models (SSMs) have emerged as a compelling alternative to attention models for long-range vision tasks, offering input-dependent recurrence with linear complexity. However, most efficient SSM variants reduce computation cost by modifying scan routes, resolutions, or traversal patterns, while largely leaving the recurrent dynamics implicit. Consequently, the model’s state-dependent memory behavior is difficult to control, particularly in compact backbones where long scan paths can exceed the effective memory horizon. We propose Token-Conditioned Poles SSM (TCP-SSM), a structured selective SSM framework that improves efficiency while making recurrence dynamics explicit and interpretable through stable poles. TCP-SSM explicitly controls SSM’s state-dependent memory be- havior using 1) real poles that model monotone or sign-alternating decay, and 2) complex-conjugate poles that capture damped oscillatory responses. Using bounded radius and angle modulation, TCP-SSM learns token-dependent poles, allowing each scan step to adapt its memory behavior to the current visual token while preserving pole stability. For practical scalability with reduced complex- ity, we integrate a lightweight low-rank input pathway, yielding an efficient scan operator that preserves linear-time scan complexity. Across image classification, semantic segmentation, and object detection, TCP-SSM reduces SSM computa- tion complexity up to 44% in Vision Mamba-style models while maintaining or surpassing baseline accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
State Space Models, Efficient Vision State Space Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NbS4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TCP-SSM, a replacement for the selective-scan operator in Vision Mamba and EfficientVMamba that is written as a transfer function with explicit poles. Each group of channels shares a small set of real poles and complex-conjugate pole pairs, whose radii and angles are bounded by sigmoid and tanh parameterisations; two scalars predicted from each token rescale all radii and angles, and a low-rank numerator drives the resulting low-order recurrence, which is run in expanded direct form. The paper argues that this preserves pole stability, makes the memory behaviour interpretable through pole radius and angle, and lowers the per-token recurrent cost from 2r+7N+12r+7N+1 to 2r+3rf+32r+3r_f+3 operations per channel. Models are trained with an L1 feature-distillation loss from the original pretrained model plus the task loss, for 600 epochs on Vim-T and 700 epochs on EfficientVMamba-T. TCP-Vim-T1 reduces SSM FLOPs by 40.6% at 76.0% top-1 against 76.1% for Vim-T, and TCP-EVM-B3 reduces them by 44% at 82.0% against 81.8% for EfficientVMamba-B; on Vim-T the paper also reports 41.1 versus 40.6 mIoU on ADE20K and 47.8 versus 45.7 box AP on COCO. The appendix adds matched-compute controls with a smaller Mamba state and end-to-end latency, throughput and memory measurements showing 1.1–1.5× speed-ups.

Critical strengths

Writing the scan as an explicitly factored, bounded, low-order recurrent filter whose radius and angle are modulated per token is a sensible parameterisation: it exposes interpretable quantities such as memory horizon and oscillation frequency, and the cost derivation in App. D is careful. The matched-compute control in App. E.2, which reduces Mamba's state size to the same SSM FLOPs under the same teacher, distillation and schedule, is the right kind of experiment for isolating the operator from the training recipe, and the appendix also reports measured end-to-end latency and memory rather than FLOPs alone.

Critical weaknesses

1. The efficiency claim is not established against the strongest simple alternatives or under a fair training setup.

By the paper's own cost model, reducing Vim's state size from N=16N=16 to N=1N=1 cuts the per-token cost from 137 to 32, about 116M SSM FLOPs for Vim-T against 295M for TCP-Vim-T1, and VMamba has shown that N=1N=1 works with little accuracy loss, yet the matched-compute control stops at N=8N=8. Token pruning and merging methods for Vision Mamba are also missing from the comparison: on Vim-T at 1.2 to 1.3 GFLOPs, DyVM [1], HiddenAlign [2] and R-MeeTo [3] reach 75.1–75.3% and MaMe [4] 76.3% after 3 to 30 epochs of fine-tuning, whereas TCP-Vim-T1 needs by my estimate about 1.3 GFLOPs and 600 epochs for 76.0%. TCP models are trained for 600 to 700 epochs, apparently from teacher-inherited weights and with feature distillation, against baselines trained for 300 epochs, so small gains may come from the schedule rather than the operator. Efficiency is argued through SSM FLOPs, a part of the whole-model cost, which is never reported. latency appears only in the appendix against the authors' own baseline with an unstated scan kernel, and several numbers contradict each other and EfficientVMamba-B configurations in Table 10 whose cost under the paper's model runs opposite to their reported FLOPs.

2. The claimed advantages of the pole formulation, stability, interpretability and novelty, are not supported.

Poles are presented as the key idea from the abstract onwards, but the paper never explains how they relate to a standard selective SSM, for instance that each diagonal entry of Mamba's Aˉ\bar A is already a real pole modulated per token through Δt\Delta_t. Propositions 1 and 2 only show Schur stability of each token's denominator with its poles frozen, while the model runs a direct-form recurrence whose coefficients change at every token; within the paper's own parameter range, with radius 0.99, base angle π/2\pi/2 and the angle scale alternating between 0.8 and 1.2, the product of two consecutive transitions has spectral radius about 1.80, and the gain normalisation cannot prevent this. Stable complex poles parameterised by radius and angle are central to S4D [5], LRU [6] and LinOSS [7], and token-dependent scalar decay to RG-LRU [8] and Mamba-2, yet none is discussed in this context, and no variant without the denominator, with real poles only or without angle modulation is reported.

References

[1] M. Wu, Z. Li, Z. Liang, et al. Dynamic Vision Mamba. arXiv:2504.04787, 2025.

[2] Z. Zhan, Z. Kong, Y. Gong, et al. Exploring Token Pruning in Vision State Space Models. NeurIPS, 2024.

[3] M. Shi, Y. Zhou, et al. Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-training. arXiv:2412.12496, 2024.

[4] J. Park, M. Son, C. Kim. Towards Efficient Vision State Space Models via Token Merging. arXiv:2508.13599, 2025.

[5] A. Gu, A. Gupta, K. Goel, C. Ré. On the Parameterization and Initialization of Diagonal State Space Models. NeurIPS, 2022.

[6] A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, S. De. Resurrecting Recurrent Neural Networks for Long Sequences. ICML, 2023.

[7] T. K. Rusch, D. Rus. Oscillatory State-Space Models. ICLR, 2025.

[8] S. De, S. L. Smith, A. Fernando, et al. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models. arXiv:2402.19427, 2024.

Questions for authors

Q1 Under the same teacher, distillation and schedule, how do conventional SSMs with N∈{1,2,4}N\in\{1,2,4\} compare with TCP-SSM in accuracy, whole-model FLOPs and measured latency? How does TCP-SSM compare with HiddenAlign, DyVM, R-MeeTo and MaMe at matched FLOPs and latency?

Q2 Which parameters of the TCP student are initialised from the teacher? Please report an original-operator student with the same initialisation, epochs and distillation, and TCP-SSM trained from scratch for 300 epochs without distillation.

Q3 Can you prove bounded-input bounded-output or exponential stability for the time-varying recurrence that is actually implemented, or report empirical evidence such as the maximum state norm along long scans and at higher resolutions?

Q4 What accuracy do the variants without the denominator, with real poles only and without angle modulation reach, and how does TCP-SSM compare with a complex-diagonal selective SSM such as LRU at the same state budget?

3 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer NbS4

We thank the reviewer for the detailed assessment and for recognizing the explicit pole formulation, matched-compute controls, computational analysis, and measured end-to-end efficiency gains. We address the concerns below.

1. Efficiency, matched training, and comparison with token-reduction methods. We first clarify that TCP's accuracy gain is not obtained by comparing a 600/700-epoch TCP model against a shorter-trained conventional baseline. Table 8 isolates the recurrent operator under matched conditions. For Vim-T, the conventional N=8N=8 student and TCP-Vim-T1 use the same pretrained teacher, feature-distillation objective, distilled layers, 600-epoch schedule, cosine scheduler, and optimizer. Likewise, EVM-T (N1,N2)=(8,10)(N_1,N_2)=(8,10) and TCP-EVM-T1 use the same 700-epoch training recipe. Their recurrent computation is also essentially identical:

\begin{array}{lccc} \hline \text{Model} & \\#\text{Param.} & \text{SSM FLOPs} & \text{Top-1} \\\\ \hline \text{Vim-T }(N=8) & 6.7\text{M} & 294.1\text{M} & 75.1 \\\\ \text{TCP-Vim-T1} & \text{7.5M} & \text{295.3M} & \text{76.0} \\\\ \text{EVM-T }(N_1=8,N_2=10) & 6.3\text{M} & 62.9\text{M} & 75.4 \\\\ \text{TCP-EVM-T1} & \text{6.4M} & \text{62.9M} & \text{76.3} \\\\ \hline \end{array}

Thus, TCP yields the same +0.9+0.9-point Top-1 gain on both backbones under matched teacher supervision, training schedule, and recurrent computation, isolating the effect of the proposed recurrence.

The initialization is matched as well: both TCP and compact conventional controls initialize all unchanged, shape-compatible backbone parameters from the same pretrained host. TCP's pole bank, modulation heads, grouped denominator, low-rank numerator, and direct path are newly initialized; in the conventional controls, only state-dependent tensors whose dimensions change are newly initialized. TCP therefore receives no initialization advantage.

The reviewer also suggests a substantially smaller conventional state dimension. This is not a matched-compute control for TCP-Vim-T1. We select N=8N=8 because its 294.1294.1M SSM FLOPs nearly exactly match TCP-Vim-T1 at 295.3295.3M.

We also compare with the cited token-reduction methods at similar whole-model compute:

MethodEfficiency mechanismTotal FLOPsTop-1HiddenAlign/ToPToken pruning + state alignment1.29G75.1DyVMDynamic token/block selection1.25G75.2R-MeeToToken merging1.28G75.3TCP-Vim-T1Recurrent-operator replacement1.2478G76.0\begin{array}{llcc} \hline \text{Method} & \text{Efficiency mechanism} & \text{Total FLOPs} & \text{Top-1} \\\\ \hline \text{HiddenAlign/ToP} & \text{Token pruning + state alignment} & 1.29\text{G} & 75.1 \\\\ \text{DyVM} & \text{Dynamic token/block selection} & 1.25\text{G} & 75.2 \\\\ \text{R-MeeTo} & \text{Token merging} & 1.28\text{G} & 75.3 \\\\ \text{TCP-Vim-T1} & \text{Recurrent-operator replacement} & \text{1.2478G} & \text{76.0} \\\\ \hline \end{array}

The initialization regime is important for interpreting this comparison. HiddenAlign, DyVM, and R-MeeTo retain the pretrained recurrent SSM, whereas TCP replaces every Vim SSM with newly initialized pole-structured dynamics. HiddenAlign modifies token pruning and hidden-state alignment; DyVM uses token/block selection; and R-MeeTo merges tokens. Their shorter optimization requirements follow from adapting a pretrained recurrence rather than relearning it.

This distinction makes the external comparison informative: despite learning a newly initialized recurrent operator, TCP-Vim-T1 outperforms all three methods at comparable total FLOPs (see table). These methods are complementary rather than architectural controls for TCP: they reduce token/block computation, whereas TCP restructures the recurrent operator applied to each retained token. The two directions can in principle be combined.

We treat MaMe separately because its FLOP accounting differs. MaMe also retains a pretrained Vim recurrence and applies token merging, but reports a different Vim-T FLOP baseline from our profiling convention and the other cited methods. We could not identify a public implementation associated with the SSM-MaMe result to reconcile the discrepancy. We therefore regard MaMe as a complementary token-merging result rather than a directly FLOP-matched recurrent-operator comparison.

Finally, our efficiency claim concerns inference efficiency: the teacher and feature-distillation loss are removed entirely at inference. SSM FLOPs isolate the operator modified by TCP, while Table 9 measures complete-model latency, throughput, and peak GPU memory under identical hardware, precision, and software settings. TCP-Vim-T1 reduces latency from 35.235.2 to 24.324.3 ms at B=128B=128, from 73.873.8 to 52.352.3 ms at B=256B=256, and from 163.4163.4 to 124.6124.6 ms at B=512B=512, with higher throughput and lower peak memory throughout. EVM-T and EVM-B show similar end-to-end improvements without a dedicated fused TCP or parallel-prefix scan kernel.

AuthorsOpenReview ↗

[2/2] Response to Reviewer NbS4

2. Initialization and distillation controls. As clarified above, no TCP-specific recurrence parameter is inherited from the teacher. The frozen teacher provides only stop-gradient feature targets in Eq. (15) and is removed at inference. Table 8 provides the original-operator student under the same initialization policy, teacher supervision, training schedule, optimizer, and recurrent-compute budget. Table 4 keeps the TCP architecture and training setting fixed while removing distillation, yielding 75.375.3\\% Top-1. Table 8 then holds teacher supervision, schedule, and recurrent computation fixed while changing the recurrence: TCP-Vim-T1 reaches 76.076.0\\%, compared with 75.175.1\\% for the conventional N=8N=8 student. Notably, TCP without distillation already exceeds the distilled compact conventional model, while identical teacher guidance increases the advantage to +0.9+0.9 points. These controls attribute the improvement to the proposed recurrence rather than distillation alone.

3. Stability of the complete token-varying recurrence. Please see our stability response above. In short terms, Proposition 2 guarantees a uniform Schur-stability margin for every token-conditioned denominator. We further show that the bounded recurrent excitation and finite-horizon gain GM(x)G_M(x) provide an explicit bound on the states and outputs of the complete token-varying scan. We additionally verify this directly on ImageNet using the realized token-conditioned denominators, where the recurrent-state and output norms remain bounded throughout the scans.

4. Pole formulation, component roles, and relation to prior recurrences. A conventional selective SSM produces token-dependent discrete poles through

pt,n=exp⁡(Δt,nan).p_{t,n}=\exp(\Delta_{t,n}a_n).

In Vim, the learned ana_n are real and negative, so pt,n∈(0,1)p_{t,n}\in(0,1): token conditioning changes the decay rate of real modes, but does not introduce signed-real responses, complex-conjugate modes, or explicit frequency control. TCP instead parameterizes the full local denominator with mixed real and complex poles,

Qt,g(z−1)=∏ℓ=1L(1−at,g,ℓz−1)∏k=1K(1−2ρt,g,kcos⁡θt,g,kz−1+ρt,g,k2z−2),Q_{t,g}(z^{-1}) = \prod_{\ell=1}^{L}(1-a_{t,g,\ell}z^{-1}) \prod_{k=1}^{K} \left( 1-2\rho_{t,g,k}\cos\theta_{t,g,k}z^{-1} +\rho_{t,g,k}^{2}z^{-2} \right),

providing direct control over both memory decay and oscillatory response while preserving the pole constraints of Proposition 2.

The distinction remains even for a complex-diagonal selective SSM. For ak=αk+jωka_k=\alpha_k+j\omega_k,

pt,k=eΔtαkejΔtωk,p_{t,k} = e^{\Delta_t\alpha_k}e^{j\Delta_t\omega_k},

so the same Δt\Delta_t jointly controls radius and angle:

log⁡ρt,k=Δtαk,θt,k=Δtωk.\log\rho_{t,k}=\Delta_t\alpha_k, \qquad \theta_{t,k}=\Delta_t\omega_k.

TCP removes this coupling through independent bounded modulation,

ρt,g,k=ρg,k,st,g(ρ),θt,g,k=clip⁡(st,g(θ)θg,k,0,π),\rho_{t,g,k} = \rho_{g,k}^{\\,s_{t,g}^{(\rho)}}, \qquad \theta_{t,g,k} = \operatorname{clip} \left( s_{t,g}^{(\theta)}\theta_{g,k},0,\pi \right),

allowing memory duration and oscillatory frequency to adapt independently to the current token.

The mixed pole bank is also order-efficient: a real pole models monotone or sign-alternating decay with one order, whereas a complex-conjugate pair uses two orders for damped oscillation. TCP therefore allocates recurrence order according to the required dynamics rather than using only real modes or unnecessarily representing every mode with a complex pair.

The denominator and numerator serve complementary roles in

Ht,g(z)=D+Pt,sc(z−1)Qt,g(z−1).H_{t,g}(z) = D+ \frac{P_{t,\mathrm{sc}}(z^{-1})} {Q_{t,g}(z^{-1})}.

Qt,gQ_{t,g} determines the recurrent memory and frequency response, while Pt,scP_{t,\mathrm{sc}} controls how token content excites these modes. Removing Qt,gQ_{t,g} removes the autoregressive recurrence itself, rather than an optional TCP component. The low-rank numerator,

Bt,i=αt,iUDiag⁡(γt)V⊤,B_{t,i} = \alpha_{t,i} U\operatorname{Diag}(\gamma_t)V^\top,

simultaneously reduces the dominant excitation cost from O(rE2)O(rE^2) to approximately O(E(r+3rf))O(E(r+3r_f)).

LRU is related through its use of stable complex diagonal eigenvalues, but its recurrence uses a single learned diagonal matrix shared across the entire sequence, i.e., static rather than token-conditioned poles. TCP instead generates token-dependent grouped denominators with mixed real/complex modes and independent radius/angle modulation, while retaining selective content-dependent excitation.

5. Table 10 configuration inconsistency. Please see our response above. The inconsistency is limited to a permutation of the TCP-EVM-B1--B5 configuration labels. The trained checkpoints, reported FLOPs and accuracies, Fig. 3, end-to-end runtime and memory measurements, and downstream results were all produced using the correct configurations and remain unchanged.

AuthorsOpenReview ↗

Follow-Up: Addressing Reviewer Concerns and Request for Reassessment

As the rebuttal discussion period approaches its conclusion, we sincerely thank the reviewer NbS4 for their constructive feedback and thoughtful engagement. We hope that our detailed responses, additional analyses, and new experimental results have adequately addressed the concerns raised. We would greatly appreciate any further feedback on outstanding questions and, if our clarifications have satisfactorily resolved the concerns, kindly invite the reviewers to reconsider their assessments and update their scores accordingly.

Reviewer_zr1xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TCP-SSM, a visual state-space model that explicitly controls recurrent dynamics through token-conditioned real and complex-conjugate poles. The method combines stable pole parameterization, grouped pole sharing, and a low-rank numerator to reduce recurrent computation. Experiments on ImageNet-1K, ADE20K, and COCO show competitive accuracy with reduced SSM FLOPs. The authors also report end-to-end inference speedups and matched-compute comparisons with conventional SSMs.

Critical strengths
  1. The paper studies the recurrent dynamics of visual SSMs rather than only modifying scan patterns. The pole-based formulation provides a useful way to control memory decay and oscillatory behavior.

  2. The efficiency improvements are supported by practical measurements. In addition to reduced SSM FLOPs, Table 9 reports end-to-end inference speedups. The matched-compute experiments in Table 8 also show that TCP-SSM performs better than simply reducing the state dimension of conventional SSMs under the same distillation settings.

Critical weaknesses
  1. Proposition 2 proves that each frozen local denominator has poles inside the unit disk. However, the actual recurrence is time-varying because its coefficients depend on the input tokens. Stability of each frozen system does not generally guarantee stability under arbitrary time-varying transitions. The paper does not establish a stability guarantee for the complete recurrence. This is an important limitation because stable and controllable memory dynamics are central claims of the paper.

  2. TCP-SSM changes several components simultaneously, including the pole structure, token-conditioned modulation, grouped sharing, and low-rank numerator. Table 5 shows that token conditioning helps, but it does not establish whether complex-conjugate poles are necessary or how much of the efficiency gain comes from the low-rank numerator. The current experiments therefore do not fully identify which design choices are responsible for the reported improvements.

Questions for authors
  1. Does Proposition 2 imply any stability guarantee for the complete time-varying recurrence? If not, can the authors clarify the scope of the stability claim and provide evidence that the recurrent states remain bounded over long sequences?

  2. Can the authors provide controlled ablations comparing real-only poles, complex-only poles, and their combination under matched computational budgets? How much of the improvement comes from the low-rank numerator rather than the pole structure?

  3. The paper reports end-to-end speedups for batch sizes 128, 256, and 512. Does TCP-SSM maintain these advantages at batch size 1, where sequential recurrence overhead may be more significant?

3 comments
AuthorsOpenReview ↗

Response to Reviewer zr1x

We thank the reviewer for the constructive assessment and for recognizing the value of explicitly modeling recurrent dynamics, the matched-compute controls, and the measured end-to-end efficiency gains.

1. Stability of the token-varying recurrence. Please see our stability response above. In short terms, Proposition 2 guarantees a uniform Schur-stability margin for every token-conditioned denominator. Combined with the bounded low-rank excitation, clipped gain normalization, and finite-horizon gain GM(x)G_M(x), this yields a finite bound on the states and outputs of the complete token-varying TCP scan. We additionally verify bounded recurrent states directly using the realized token-conditioned dynamics on ImageNet.

2. Component roles and existing ablations.

The manuscript already contains several complementary ablations that isolate the principal design choices. Table 4 studies the effect of feature-level distillation; Table 5 compares static poles, shared token conditioning, and group-specific token conditioning; Table 8 compares TCP against compact conventional SSMs under identical distillation and matched recurrent computation; Table 11 varies the number of denominator-sharing groups; and Fig. 3 varies the recurrent capacity over a broad accuracy--computation range, reaching a 70.670.6\\% reduction in SSM FLOPs.

The real and complex poles serve complementary dynamical roles. A real pole produces aka^k, capturing monotone or sign-alternating decay, whereas a complex-conjugate pair produces

ρk![c1cos⁡(kθ)+c2sin⁡(kθ)],\rho^k\\!\left[c_1\cos(k\theta)+c_2\sin(k\theta)\right],

capturing damped oscillatory and frequency-selective behavior. Their combination therefore represents both nonoscillatory and oscillatory memory without using unnecessary recurrence order.

The denominator and numerator likewise play distinct roles:

Ht,g(z)=D+Pt,sc(z−1)Qt,g(z−1).H_{t,g}(z)=D+\frac{P_{t,\mathrm{sc}}(z^{-1})}{Q_{t,g}(z^{-1})}.

Qt,gQ_{t,g} determines the poles, and hence the recurrent memory and frequency response, while the numerator controls how token information excites those dynamics. Setting Qt,g=1Q_{t,g}=1 removes the autoregressive feedback entirely and reduces the operator to a finite-window feed-forward mixer rather than a recurrent SSM. Therefore, a "TCP without the denominator" model would no longer represent the recurrent system that TCP is designed to parameterize. The low-rank numerator addresses efficiency rather than stability: replacing dense lag-dependent E×EE\times E maps with

Bt,i=αt,iUDiag⁡(γt)V⊤B_{t,i}=\alpha_{t,i}U\operatorname{Diag}(\gamma_t)V^\top

reduces the dominant numerator cost from O(rE2)O(rE^2) to approximately O(E(r+3rf))O(E(r+3r_f)). Thus, the pole-structured denominator provides explicit recurrent dynamics, while the low-rank numerator provides efficient content-dependent excitation. The existing ablations separately examine token conditioning, grouping, distillation, recurrent capacity, and matched-compute behavior, while the full model evaluates their combined accuracy--efficiency benefit.

3. Batch-size-1 inference. The recurrent FLOP reduction is independent of batch size because both operators share the factor BMEBME:

FTCP=BME,cTCP,FVim=BME,cVim,FTCPFVim=cTCPcVim.F_{\mathrm{TCP}}=BME\\,c_{\mathrm{TCP}},\qquad F_{\mathrm{Vim}}=BME\\,c_{\mathrm{Vim}},\qquad \frac{F_{\mathrm{TCP}}}{F_{\mathrm{Vim}}}=\frac{c_{\mathrm{TCP}}}{c_{\mathrm{Vim}}}.

Table 9 already reports full-model end-to-end measurements at batch sizes 128128, 256256, and 512512: TCP-Vim-T1 achieves 1.3×1.3\times--1.5×1.5\times speedup, TCP-EVM-T1 1.17×1.17\times--1.21×1.21\times, and TCP-EVM-B3 1.11×1.11\times--1.16×1.16\times, with lower peak memory throughout. These gains are obtained without an associative parallel-prefix scan or hardware-specific scheduling, showing that the reduced recurrent computation already translates into practical full-model acceleration. We will also add the speed results for batch-1 shortly.

AuthorsOpenReview ↗

Additional Small-Batch Inference Results

To further address the reviewer's concern regarding practical inference efficiency at small batch sizes, we conducted additional end-to-end measurements comparing TCP-EVM-B3 against the original EVM-B at batch sizes 1, 8, and 16. These results complement the larger-batch evaluations in Table 9 and provide a more comprehensive assessment of TCP's practical efficiency across different inference settings. The complete results are summarized below.

End-to-End Inference Efficiency. Lower latency and peak memory, and higher throughput, are better.

Model PairBBase msTCP msSpeedupBase img/sTCP img/sBase MiBTCP MiBVim-T / TCP-Vim-T112835.224.31.5×3635.35259.5370.8335.225673.852.31.4×3469.24894.1695.1616.5512163.4124.61.3×3134.24109.01342.91176.7EVM-T / TCP-EVM-T112822.318.41.2×5730.46958.9377.9323.425650.642.11.2×5057.16082.7706.4692.4512110.094.41.2×4652.85420.9926.9874.4EVM-B / TCP-EVM-B311.4031.1021.27×712.91907.05170.64163.6582.5912.4801.04×3087.953225.56330.18268.47165.2904.4701.18×3024.703579.64514.23393.1212859.050.81.2×2170.52521.8724.3669.2256131.3116.01.1×1950.32207.11294.51184.0512266.6240.51.1×1920.72129.01364.51298.2\begin{array}{lrrrrrrrr} \hline \text{Model Pair} & B & \text{Base ms} & \text{TCP ms} & \text{Speedup} & \text{Base img/s} & \text{TCP img/s} & \text{Base MiB} & \text{TCP MiB} \\\\ \hline \text{Vim-T / TCP-Vim-T1} & 128 & 35.2 & 24.3 & 1.5\times & 3635.3 & 5259.5 & 370.8 & 335.2 \\\\ & 256 & 73.8 & 52.3 & 1.4\times & 3469.2 & 4894.1 & 695.1 & 616.5 \\\\ & 512 & 163.4 & 124.6 & 1.3\times & 3134.2 & 4109.0 & 1342.9 & 1176.7 \\\\ \hline \text{EVM-T / TCP-EVM-T1} & 128 & 22.3 & 18.4 & 1.2\times & 5730.4 & 6958.9 & 377.9 & 323.4 \\\\ & 256 & 50.6 & 42.1 & 1.2\times & 5057.1 & 6082.7 & 706.4 & 692.4 \\\\ & 512 & 110.0 & 94.4 & 1.2\times & 4652.8 & 5420.9 & 926.9 & 874.4 \\\\ \hline \text{EVM-B / TCP-EVM-B3} & 1 & 1.403 & 1.102 & 1.27\times & 712.91 & 907.05 & 170.64 & 163.65 \\\\ & 8 & 2.591 & 2.480 & 1.04\times & 3087.95 & 3225.56 & 330.18 & 268.47 \\\\ & 16 & 5.290 & 4.470 & 1.18\times & 3024.70 & 3579.64 & 514.23 & 393.12 \\\\ & 128 & 59.0 & 50.8 & 1.2\times & 2170.5 & 2521.8 & 724.3 & 669.2 \\\\ & 256 & 131.3 & 116.0 & 1.1\times & 1950.3 & 2207.1 & 1294.5 & 1184.0 \\\\ & 512 & 266.6 & 240.5 & 1.1\times & 1920.7 & 2129.0 & 1364.5 & 1298.2 \\\\ \hline \end{array}

Analysis of Small-Batch Inference Efficiency. Notably, TCP-EVM-B3 achieves a 1.27$\times$ speedup even at B=1B=1, reducing end-to-end latency from 1.403 to 1.102 ms (21.5%) while increasing throughput from 712.91 to 907.05 images/s (27.2%). Peak GPU memory also decreases from 170.64 to 163.65 MiB. At batch sizes 8 and 16, TCP maintains lower inference latency, achieving 1.04$\times$ and 1.18$\times$ speedups, respectively, alongside reduced peak memory. These results demonstrate that TCP's computational savings translate into practical acceleration even when batch-level parallelism is limited.

Together with the larger-batch measurements, our results consistently demonstrate lower end-to-end latency, higher throughput, and reduced peak GPU memory across all evaluated settings. Importantly, these gains are achieved without a dedicated associative parallel-prefix scan kernel. This provides additional empirical evidence that TCP's efficiency improvements extend beyond theoretical SSM FLOP reductions and remain effective for single-image inference.

AuthorsOpenReview ↗

Follow-Up: Addressing Reviewer Concerns and Request for Reassessment

As the rebuttal discussion period approaches its conclusion, we sincerely thank the reviewer zr1x for their constructive feedback and thoughtful engagement. We hope that our detailed responses, additional analyses, and new experimental results have adequately addressed the concerns raised. We would greatly appreciate any further feedback on outstanding questions and, if our clarifications have satisfactorily resolved the concerns, kindly invite the reviewers to reconsider their assessments and update their scores accordingly.

Reviewer_AvU7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TCP-SSM, a visual state space operator that parameterizes recurrent dynamics through token-conditioned real and complex-conjugate poles. The method combines grouped pole sharing with a low-rank input pathway to reduce computation and uses feature-level distillation from the original model during training. Experiments on image classification, semantic segmentation, and object detection report reduced SSM computation while maintaining comparable or improved accuracy. Additional evaluations show end-to-end inference efficiency gains over the evaluated baselines.

Critical strengths
  1. The pole-based formulation improves interpretability. The explicit pole parameterization offers a useful interpretation of how the model retains and forgets information, with pole magnitudes controlling decay and pole angles determining oscillatory behavior.

  2. Ablation experiments support the architectural contribution. Under identical distillation settings and comparable SSM computation, TCP improves Top-1 accuracy on both Vim-T and EVM-T. These results suggest that the gains are not attributable to distillation alone.

  3. Evaluation across tasks with practical efficiency gains. Experiments cover image classification, semantic segmentation, and object detection. End-to-end measurements further demonstrate improvements in latency, throughput, and peak memory usage.

Critical weaknesses
  1. The stability guarantee is limited to frozen local systems. Proposition 2 and Appendix C establish stability when the coefficients are fixed, whereas Eq. (14) defines a time-varying recurrence. These local guarantees alone do not establish stability of the time-varying scan. The authors should clarify whether additional conditions are needed to extend the guarantee to the full recurrence.

  2. The reported configurations appear inconsistent with the computational results. Table 10 assigns larger recurrence orders and numerator ranks to TCP-EVM-B2 than to TCP-EVM-B5, with identical group counts. Under the stated backbone and replacement settings, Eq. (44) implies higher computation for B2. However, the reported SSM computation reductions are 62.3% for B2 and approximately 0.7% for B5.

  3. Evidence for improved long-range memory remains indirect. Appendix E provides useful visualizations of pole-derived timescales and frozen local impulse responses. However, these analyses do not directly establish better retention of distant information during the actual time-varying scan.

Questions for authors

1.Does the proposed parameterization guarantee stability of the time-varying recurrence, or only of each frozen local system? If the former, what additional conditions establish this guarantee?

2.Could the authors reconcile the configurations in Table 10 with the reported FLOPs, particularly for TCP-EVM-B2 and TCP-EVM-B5?

3.Could the authors provide a controlled comparison of distant-token influence or performance across sequence lengths to assess whether TCP improves long-range memory over the baseline?

4.What additional training time and peak memory does feature-level distillation require under matched hardware conditions? Do the inference gains in Table 9 also hold at small batch sizes?

5.How does TCP compare with relevant visual linear-attention methods[1,2] under comparable total model computation or measured latency? It would help establish its broader accuracy–efficiency trade-off.

[1] Demystify Mamba in Vision: A Linear Attention Perspective[C]. NeurIPS 2024.

[2] EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction[C], ICCV 2023.

3 comments
AuthorsOpenReview ↗

Response to Reviewer AvU7

We thank the reviewer for the constructive feedback and for recognizing our interpretable pole formulation, matched-compute controls, and practical efficiency gains. We address the concerns below.

1. Stability of the token-varying recurrence. As detailed in our shared stability response above, Proposition 2 guarantees a uniform Schur-stability margin for each token-conditioned denominator. To address the complete time-varying recurrence, we also bound its excitation and define the finite-horizon gain GM(x)G_M(x) over the full coefficient trajectory, yielding explicit bounds on recurrent states and outputs for each realized scan. ImageNet evaluation with realized token-conditioned denominators further confirms bounded state and output norms.

2. Table 10 configuration inconsistency. As noted in our shared response above, the TCP-EVM-B1--B5 labels in Table 10 were permuted. The checkpoints and all reported accuracy, FLOPs, runtime, memory, downstream results, and Fig. 3 remain unchanged. We will correct Table 10 in the revision.

3. Long-range memory and distant-token retention. TCP aims for adaptive, not uniformly longer, memory: relevant history can persist while irrelevant information decays. For a pole with radius ρ\rho, the response envelope is ρk\rho^k and the e-folding horizon is τ=−1/log⁡ρ\tau=-1/\log\rho. By Eq. (9), token conditioning gives

ρt,g,k=ρg,kst,g(ρ),τt,g,k=τg,kbasest,g(ρ).\rho_{t,g,k}=\rho_{g,k}^{s_{t,g}^{(\rho)}},\qquad \tau_{t,g,k}=\frac{\tau_{g,k}^{\mathrm{base}}}{s_{t,g}^{(\rho)}}.

Thus, st,g(ρ)<1s_{t,g}^{(\rho)}<1 extends memory, while st,g(ρ)>1s_{t,g}^{(\rho)}>1 accelerates forgetting; angle modulation separately controls oscillation. Figures 1 and 4--5 visualize these learned timescales and impulse responses. Consistent with this mechanism, Table 5 shows that token-conditioned poles outperform static poles (76.0%--76.1% versus 75.4% Top-1), while Table 8 reports a +0.9-point gain over compact conventional SSMs at matched recurrent compute. Following visual-SSM evaluation protocols for memory effectiveness, we test spatial modeling on ADE20K (5122512^2) and COCO (102421024^2): TCP improves Vim-T from 40.6% to 41.1% mIoU and from 45.7 to 47.8 box AP, respectively, with 40.6% fewer SSM FLOPs. These results support effective high-resolution spatial modeling without uniformly persistent memory.

4. Feature-distillation overhead and small-batch inference. Distillation is confined to training: the frozen, stop-gradient teacher adds one forward pass and the feature loss in Eq. (15), but no teacher backward pass or optimizer states; the teacher is absent at inference. Crucially, Table 8's compact conventional controls use the same teacher, distilled layers, optimizer, and training schedule, isolating the recurrence design. At matched recurrent compute, TCP improves Top-1 from 75.1% to 76.0% on Vim-T and from 75.4% to 76.3% on EVM-T.

The theoretical recurrent-FLOP ratio is independent of batch size because both operators scale with BMEBME:

FTCP=BME,cTCP,FVim=BME,cVim,FTCPFVim=cTCPcVim.F_{\mathrm{TCP}}=BME\\,c_{\mathrm{TCP}},\quad F_{\mathrm{Vim}}=BME\\,c_{\mathrm{Vim}},\quad \frac{F_{\mathrm{TCP}}}{F_{\mathrm{Vim}}}=\frac{c_{\mathrm{TCP}}}{c_{\mathrm{Vim}}}.

Beyond FLOPs, Table 9 reports full-model end-to-end speedups of 1.3--1.5$\times$ (TCP-Vim-T1), 1.17--1.21$\times$ (TCP-EVM-T1), and 1.11--1.16$\times$ (TCP-EVM-B3), with lower peak memory throughout, under identical hardware and inference settings. These gains require neither a parallel-prefix scan nor hardware-specific scheduling. We will also provide batch-1 speed measurements shortly.

5. Comparison with visual linear-attention methods. The cited methods provide useful context but redesign the visual mixer or backbone, whereas TCP replaces only the SSM operator while preserving the original architecture. Thus, paired Vim/TCP and EVM/TCP comparisons isolate the proposed recurrence, with all non-SSM computation unchanged. Table 9 further confirms that the SSM-FLOP reductions translate into lower end-to-end latency, higher throughput, and reduced peak memory under identical conditions.

AuthorsOpenReview ↗

Additional Batch-Size-1 Inference Results

To further address the reviewer's concern regarding practical inference efficiency at small batch sizes, we conducted additional end-to-end measurements comparing TCP-EVM-B3 against the original EVM-B at batch sizes 1, 8, and 16. These results complement the larger-batch evaluations in Table 9 and provide a more comprehensive assessment of TCP's practical efficiency across different inference settings. The complete results are summarized below.

End-to-End Inference Efficiency. Lower latency and peak memory, and higher throughput, are better.

Model PairBBase msTCP msSpeedupBase img/sTCP img/sBase MiBTCP MiBVim-T / TCP-Vim-T112835.224.31.5×3635.35259.5370.8335.225673.852.31.4×3469.24894.1695.1616.5512163.4124.61.3×3134.24109.01342.91176.7EVM-T / TCP-EVM-T112822.318.41.2×5730.46958.9377.9323.425650.642.11.2×5057.16082.7706.4692.4512110.094.41.2×4652.85420.9926.9874.4EVM-B / TCP-EVM-B311.4031.1021.27×712.91907.05170.64163.6582.5912.4801.04×3087.953225.56330.18268.47165.2904.4701.18×3024.703579.64514.23393.1212859.050.81.2×2170.52521.8724.3669.2256131.3116.01.1×1950.32207.11294.51184.0512266.6240.51.1×1920.72129.01364.51298.2\begin{array}{lrrrrrrrr} \hline \text{Model Pair} & B & \text{Base ms} & \text{TCP ms} & \text{Speedup} & \text{Base img/s} & \text{TCP img/s} & \text{Base MiB} & \text{TCP MiB} \\\\ \hline \text{Vim-T / TCP-Vim-T1} & 128 & 35.2 & 24.3 & 1.5\times & 3635.3 & 5259.5 & 370.8 & 335.2 \\\\ & 256 & 73.8 & 52.3 & 1.4\times & 3469.2 & 4894.1 & 695.1 & 616.5 \\\\ & 512 & 163.4 & 124.6 & 1.3\times & 3134.2 & 4109.0 & 1342.9 & 1176.7 \\\\ \hline \text{EVM-T / TCP-EVM-T1} & 128 & 22.3 & 18.4 & 1.2\times & 5730.4 & 6958.9 & 377.9 & 323.4 \\\\ & 256 & 50.6 & 42.1 & 1.2\times & 5057.1 & 6082.7 & 706.4 & 692.4 \\\\ & 512 & 110.0 & 94.4 & 1.2\times & 4652.8 & 5420.9 & 926.9 & 874.4 \\\\ \hline \text{EVM-B / TCP-EVM-B3} & 1 & 1.403 & 1.102 & 1.27\times & 712.91 & 907.05 & 170.64 & 163.65 \\\\ & 8 & 2.591 & 2.480 & 1.04\times & 3087.95 & 3225.56 & 330.18 & 268.47 \\\\ & 16 & 5.290 & 4.470 & 1.18\times & 3024.70 & 3579.64 & 514.23 & 393.12 \\\\ & 128 & 59.0 & 50.8 & 1.2\times & 2170.5 & 2521.8 & 724.3 & 669.2 \\\\ & 256 & 131.3 & 116.0 & 1.1\times & 1950.3 & 2207.1 & 1294.5 & 1184.0 \\\\ & 512 & 266.6 & 240.5 & 1.1\times & 1920.7 & 2129.0 & 1364.5 & 1298.2 \\\\ \hline \end{array}

Analysis of Small-Batch Inference Efficiency. Notably, TCP-EVM-B3 achieves a 1.27$\times$ speedup even at B=1B=1, reducing end-to-end latency from 1.403 to 1.102 ms (21.5%) while increasing throughput from 712.91 to 907.05 images/s (27.2%). Peak GPU memory also decreases from 170.64 to 163.65 MiB. At batch sizes 8 and 16, TCP maintains lower inference latency, achieving 1.04$\times$ and 1.18$\times$ speedups, respectively, alongside reduced peak memory. These results demonstrate that TCP's computational savings translate into practical acceleration even when batch-level parallelism is limited.

Together with the larger-batch measurements, our results consistently demonstrate lower end-to-end latency, higher throughput, and reduced peak GPU memory across all evaluated settings. Importantly, these gains are achieved without a dedicated associative parallel-prefix scan kernel. This provides additional empirical evidence that TCP's efficiency improvements extend beyond theoretical SSM FLOP reductions and remain effective for single-image inference.

AuthorsOpenReview ↗

Follow-Up: Addressing Reviewer Concerns and Request for Reassessment

As the rebuttal discussion period approaches its conclusion, we sincerely thank the reviewer AvU7 for their constructive feedback and thoughtful engagement. We hope that our detailed responses, additional analyses, and new experimental results have adequately addressed the concerns raised. We would greatly appreciate any further feedback on outstanding questions and, if our clarifications have satisfactorily resolved the concerns, kindly invite the reviewers to reconsider their assessments and update their scores accordingly.

Comments on the paper

2 comments
AuthorsOpenReview ↗

[1/2] Shared Response to all the reviewers: Stability of the Complete Token-Varying TCP Recurrence

We thank the reviewers for highlighting the distinction between local pole stability and the behavior of the full time-varying recurrence (Eq. 14). While Proposition 2 guarantees the former, we address the latter by analyzing the complete finite-horizon scan.

Specifically, for every token tt and group gg, Proposition 2 guarantees

∣at,g,ℓ∣≤1−ϵρ,0<ρt,g,k≤1−ϵρ,ϵρ>0,|a_{t,g,\ell}|\le1-\epsilon_\rho,\qquad0<\rho_{t,g,k}\le1-\epsilon_\rho,\qquad\epsilon_\rho>0,

so each Qt,gQ_{t,g} has poles strictly inside the unit circle. Since these denominators vary across tokens, we further consider the complete recurrence:

yt(g)=η~t(g)−∑i=1rqt,g,i(x)yt−i(g).y_t^{(g)}=\widetilde{\eta}_t^{(g)}-\sum_{i=1}^{r}q_{t,g,i}(x)y_{t-i}^{(g)}.

Its excitation is finite on each realized scan due to normalized backbone features, the finite rr-token numerator window, and bounded gating γt=σ(Wγxt)\gamma_t=\sigma(W_\gamma x_t). Eq. (47) additionally normalizes recurrent gain, clipped to 0.5≤ωt,g≤20.5\le\omega_{t,g}\le2. Consequently, ηmax⁡(x)=max⁡t,g∥η~t(g)∥∞<∞\eta_{\max}(x)=\max_{t,g}\lVert\widetilde{\eta}_t^{(g)}\rVert_\infty<\infty.

To account for the entire coefficient trajectory, rather than a frozen denominator, we define

ht,k(g)(x)={0,t<k,1,t=k,−∑i=1rqt,g,i(x)ht−i,k(g)(x),t>k.h_{t,k}^{(g)}(x)=\begin{cases}0,&t<k,\\\\1,&t=k,\\\\-\displaystyle\sum_{i=1}^{r}q_{t,g,i}(x)h_{t-i,k}^{(g)}(x),&t>k.\end{cases}

This yields yt(g)=∑k=1tht,k(g)(x)η~k(g)y_t^{(g)}=\sum_{k=1}^{t}h_{t,k}^{(g)}(x)\widetilde{\eta}_k^{(g)}, allowing us to characterize the finite-horizon gain as

GM(x)=max⁡gmax⁡t≤M∑k=1t∣ht,k(g)(x)∣.G_M(x)=\max_g\max_{t\le M}\sum_{k=1}^{t}|h_{t,k}^{(g)}(x)|.

Since MM and the coefficients are finite, GM(x)<∞G_M(x)<\infty. Therefore, with ot=yt+Dxto_t=y_t+Dx_t,

max⁡t,g∥yt(g)∥∞≤GM(x)ηmax⁡(x),max⁡t≤M∥ot∥∞≤GM(x)ηmax⁡(x)+max⁡t≤M∥Dxt∥∞<∞.\max_{t,g}\lVert y_t^{(g)}\rVert_\infty\le G_M(x)\eta_{\max}(x),\qquad\max_{t\le M}\lVert o_t\rVert_\infty\le G_M(x)\eta_{\max}(x)+\max_{t\le M}\lVert Dx_t\rVert_\infty<\infty.

Thus, the bound applies to the complete realized token-varying scan, rather than only its individual denominators.

To complement this analysis, we evaluated the trained TCP-Vim-T1 checkpoint on 1000 ImageNet samples using the actual token-conditioned coefficients, obtaining

max⁡x,t,g∥yt(g)(x)∥∞=110.99,max⁡x,t∥ot(x)∥∞=110.39,\max_{x,t,g}\lVert y_t^{(g)}(x)\rVert_\infty=110.99,\qquad\max_{x,t}\lVert o_t(x)\rVert_\infty=110.39,

providing empirical evidence that the states and outputs remain numerically controlled over the evaluated scans.

Finally, regarding Reviewer NbS4's two-step construction, a spectral radius above one for a short transition product permits transient amplification but does not, by itself, establish instability of the complete finite realized recurrence. Our GM(x)G_M(x) characterizes its cumulative finite-horizon amplification, while the observed maxima verify bounded behavior under the actual token-conditioned dynamics. These findings are consistent with Proposition 2 without asserting uniform stability under arbitrary switching.

AuthorsOpenReview ↗

[2/2] Shared Response to all the reviewers: Correction of Table 10 Configurations.

We thank Reviewers AvU7 and NbS4 for identifying the inconsistency in Table 10. This was a labeling error in the TCP-EVM-B1--B5 configuration rows: the configuration labels were permuted, while the trained checkpoints and all reported accuracy, FLOPs, runtime, memory, and downstream results were obtained with the correct configurations and therefore remain unchanged.

The corrected configurations are

ModelStageLKrrfB1S12148S221410B2S122610S222612B3S132716S242820B4S1451422S2451424B5S1661826S2661828\begin{array}{c|ccccc} \text{Model} & \text{Stage} & L & K & r & r_f \\\\ \hline \text{B1} & \text{S1} & 2 & 1 & 4 & 8 \\\\ & \text{S2} & 2 & 1 & 4 & 10 \\\\ \text{B2} & \text{S1} & 2 & 2 & 6 & 10 \\\\ & \text{S2} & 2 & 2 & 6 & 12 \\\\ \text{B3} & \text{S1} & 3 & 2 & 7 & 16 \\\\ & \text{S2} & 4 & 2 & 8 & 20 \\\\ \text{B4} & \text{S1} & 4 & 5 & 14 & 22 \\\\ & \text{S2} & 4 & 5 & 14 & 24 \\\\ \text{B5} & \text{S1} & 6 & 6 & 18 & 26 \\\\ & \text{S2} & 6 & 6 & 18 & 28 \end{array}

This ordering is consistent with the cost model in Eq. (44), whose dominant per-token-channel term is the recurrent capacity and computation increase monotonically from B1 to B5, matching the accuracy--efficiency trend in Fig. 3. We will correct the labels in Table 10 in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.