acceptodds
Under review as a conference paper at ICLR 2027

Recurrent Vision in Parallel: Spectral Convolutional State Space Models

Abstract

Brains and deep neural networks share the goal of building rich and robust visual representations but pursue it through different means. Cortical circuits route sensory signals through re-entrant feedback, refining their representations iteratively over time. Deep neural networks (DNNs), by contrast, refine theirs across layers of distinct parameters, in a single feedforward pass. This distinction becomes especially pronounced for high-dimensional signals like video, where capturing rich spatiotemporal structure either demands many sequential refinement steps or a rapid growth in model size to encode it all at once. Brains take the first route via recurrence, but its sequential nature does not translate easily to GPUs. Here, we close this gap with a parallelizable state space model (SSM) formulation that lifts convolutional recurrence into the spectral domain, where composing convolutions reduces to pointwise multiplication and yields linear-time scaling. We instantiate this idea as spectral convolutional state space models (sCSSMs): a drop\-in primitive that composes with other SSM circuits including Mamba and Gated Delta Networks. A single sCSSM layer rivals human accuracy on visual reasoning challenges where transformers fail, and reproduces the time–accuracy tradeoff that humans display on these tasks. Replacing self-attention with sCSSMs on ImageNet-1K and as a LoRA adapter on a video generation model beats Standard and Transformer baselines with fewer parameters on VideoPhy-2.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
convolution, state space model, SSM, video modeling, computer vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_dqoYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes spectral convolutional state space models (sCSSMs) for parallelizing spatially coupled recurrence. The main idea is to represent circulant convolution operators in the Fourier domain, where recurrent kernel composition becomes pointwise multiplication. The authors derive the corresponding SSM formulation and show that it supports parallel scans. Experiments include runtime benchmarks, PathFinder and PathTracker, ImageNet classification, and LoRA adaptation of a video generation model.

Critical strengths

The paper addresses a concrete limitation of convolutional SSMs: spatially coupled recurrence is expensive to parallelize. The spectral formulation provides a reasonable solution under circulant boundary conditions, and the runtime experiments show clear improvements over the spatial implementation. The controlled visual reasoning experiments are also useful, particularly the ablations removing spatial or temporal recurrence.

Critical weaknesses

Section 4 develops a general spectral SSM with channel mixing within each frequency block. However, Appendix F states that the actual implementation uses depthwise spatial kernels, reducing these blocks to scalar transitions. The more general formulation, including the proposed change of basis for channel mixing, is therefore not evaluated. This makes it difficult to assess how much of the theoretical framework is practically useful beyond the depthwise case.

The strongest results are obtained on PathFinder and PathTracker, where spatially local recurrence is particularly well suited to the task. On ImageNet, the comparison is limited to a small SHViT backbone, and increasing the number of recurrent steps also increases computation. The paper mainly reports accuracy against parameter count rather than a controlled accuracy-latency comparison.

The video generation results are less convincing. Appendix F reports only a 1–2 percentage point improvement on VideoPhy-2 over the original CogVideoX model, while training is approximately 2.4 times slower than the other LoRA variants. All reported results use a single random seed. Without uncertainty estimates, it is difficult to judge whether this relatively small improvement is reliable.

The theoretical result concerns the work and parallel depth of the recurrent scan, not the complete model. FFT operations, complex-valued intermediate states, and additional transformations also contribute to runtime and memory usage. Although the single-layer benchmarks are useful, they do not establish that the proposed method is more efficient in end-to-end vision applications. The video adaptation experiment, in fact, shows a substantial training-time overhead.

Questions for authors
  1. Why is the full channel-mixing formulation not evaluated? Does it provide any benefit over the depthwise implementation, and how does its computational cost compare?

  2. Can the authors provide ImageNet results under matched inference latency or FLOPs, rather than only matched parameter counts?

  3. Is the 1–2 percentage point improvement on VideoPhy-2 consistent across random seeds? How does the method compare when the additional training time is taken into account?

  4. How much of the reported runtime advantage remains when FFT overhead and the complete model are included?

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer dqoY

We thank the reviewer for the clear and fair assessment. The weaknesses listed are limitations we report in App. F, as the reviewer notes, and we agree they should carry more weight in the main text.

W1 / Q1: full channel mixing. All experiments use a depthwise (per-channel) transition. This is stated in App. E.2 and App. F but not in the main text, which is our error; Sec. 5 will state it and the "no compromises" sentence in Sec. 4.1 will be reworded. Sec. 4 shows that a channel-mixing transition can be brought to diagonal form by absorbing a per-frequency change of basis Q_c into the input and output maps. We do not evaluate that general model, and the input and output maps of our implementation are not per-frequency, so we do not claim the evaluated model is equivalent to it. What we evaluate is the depthwise subclass, with channel mixing done outside the recurrence (stem, output projection, and the Q/K/V projections of the GDN variant). All empirical claims are for that subclass, which is the more constrained model. The general form would add a c x c eigendecomposition per frequency, O(hw c^3), and requires each block to be diagonalizable and well conditioned; we will state these costs and assumptions. We have not measured whether channel mixing in the recurrence helps.

W2 / Q2: ImageNet under matched compute. We do not have matched-latency results. Repeating the input T times runs the sCSSM mixer on T frames, so the T=8 model costs more than SHViT-S1 at inference, and parameter count is not a proxy for compute. We will report FLOPs, throughput and peak memory for every variant and limit the claim to parameter efficiency. Relative to our Hybrid SHViT-S1 control, which is trained with the same recipe (73.78%), the GDN-sCSSM gains 0.84 points at T=1 (74.62%) and a further 0.41 points at T=8 (75.03%).

W3 / Q3: VideoPhy-2. The GDN-sCSSM adapter is numerically better than the base model (147 vs 142 Joint successes out of 591 prompts; 24.9% vs 24.0%), but the difference is not statistically significant (paired McNemar test, p = 0.46), and App. F already describes the gain as modest. We did not run multiple seeds: App. E.4 reports a single seed, each adapter needs about 22 GPU-hours of generation per evaluation, and we did not have the compute to train and score enough independent models for statistical power. We will add the test results to Sec. 5.4 and soften the claims: the abstract and Sec. 5.4 will no longer say that the adapter "beats" the others or "corrects" physical errors. The sCSSM adapter is 2.4x slower per training step (App. F), and we make no claim at matched training time.

W4 / Q4: end-to-end cost. The step times in Fig. 2 are measured forward plus backward and include the FFT and inverse FFT. Prop. 3 covers only the scan, and we will add the rest: the FFTs add O(T c hw log(hw)) work with depth independent of T, and complex states double per-block memory (App. F). The reviewer is right that a single-layer benchmark does not establish end-to-end efficiency. App. C says this, and the only end-to-end timing we have, the LoRA, is slower. The revision moves both statements into Sec. 5.1.

Reviewer_SwQqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  1. This paper investigates how to effectively parallelize spatially interactive recurrent vision models on GPUs to avoid the increased representation and computational costs associated with directly combining spatial convolutional kernels.

  2. The authors propose spectral convolutional state space models (sCSSMs): using convolutional structures with recurrent boundaries and Fourier transforms, spatial convolutional recurrents are converted into linear recursions in frequency patterns, and then associative scans are used for time-parallel processing. The derivation includes channel mixing, and the actual evaluation kernel employs depthwise spatial recursion and frequency domain stabilization, combined with a Mamba and Gated Delta Network (GDN) design.

  3. The core contribution is implementing recurrent convolutions with a computable frequency domain representation, demonstrating its applicability in LoRA fine-tuning of PathFinder, PathTracker, ImageNet-1K, and CogVideoX. Competitive accuracy and efficiency are achieved in control tasks, but "local recurrent equivalence," "general composability," and practical end-to-end efficiency still require more precise proof and comparison.

Critical strengths
  1. The problem and computational motivation are clearly defined: Sections 3 and 4 clearly address the structural conflict between spatial convolutional loops and parallel scans, and utilize Fourier field representation to avoid explicitly storing the dilated convolutional kernel. This is a concrete and valuable computational problem.

  2. The actual depthwise kernel has a concise mathematical form: Equation (19) in Appendix E.2 is a complex scalar recursion independent of each frequency and channel, which can indeed be used for associative scans using affine combinations; Equation (20) also provides a clear contraction bound for this constrained scalar kernel.

  3. The mechanism testing includes useful controls: Figure 2 separates the stepwise spatial loop, spatial domain parallel, and frequency domain versions, and takes the median of 100 timings after JIT, which helps to pinpoint the sources of computational cost.

  4. The experiments cover a variety of applications: In addition to synthetic perception tasks, the parameters of ImageNet and pre-trained video generators are evaluated for efficient tuning, including spatial, temporal, and convolutional Hybrid SHViT controls, which is superior to reporting only a single application.

  5. The training and evaluation process is transparent to a certain extent: Section 5.2 explains early stopping on the validation set and retaining the test set for evaluation, and Appendix E provides information on hardware, accuracy, training configuration, and video generation procedures; this information provides a foundation for reproducible studies.

Critical weaknesses
  1. Stabilization and Discretization in Practice May Disrupt the Rigorous Local Cyclic Interpretation.
    The Fourier transform itself does not change the locality of the convolution operator, but after applying nonlinear amplitude scaling to each frequency in Equation (20), the effective core obtained by its inverse transform generally no longer retains the original k×k finite support. The text also acknowledges that continuous-time discretization may produce global support. Therefore, the actual results cannot be directly interpreted as only local information gradually propagating, which affects the core argument of this paper connecting cortical local connections and perceptual grouping. Stability also comes from contraction parameterization, rather than being automatically obtained simply by switching to the frequency domain.

  2. Substantial Inconsistencies Exist Between Complexity Propositions and Operator Definitions. Proposition 1's O(T⁴) is used to analyze a certain implementation under the extended core and direct convolution cost model, and cannot yet be considered a necessary lower bound for all spatial ConvSSMs. Appendice G.3 uses cyclic modulo for image indexing, yet it also causes image support to expand with the combination, resulting in inconsistencies between the two boundary assumptions. Proposition 2 claims O(hwc) as a constant, but Appendice G.4 only reduces it to a general hwc-dimensional linear system, and cannot directly derive linear dimension costs. Eq.s (26) and (41) also write the composition of general matrices/block matrices as element-wise multiplication; this only applies to the case of diagonalization.

  3. Omission of Directly Related Structured Convolution (SSM) and Frequency Domain Vision Research. ConvT3 (ICLR 2026) has implemented a non-1×1 3×3 state core through a restricted tridiagonal Toeplitz structure, diagonalization, and parallel scan, which is highly relevant to the problem and solution presented in this paper, yet no comparison has been found. GFNet, FNO, and AFNO have also demonstrated frequency-domain global convolution, channel structure, and efficient visual token mixing. This does not mean that this paper lacks novelty, but it is currently insufficient to consider FFT or breaking the 1×1 constraint as a complete basis for novelty.

  4. Claims of general channel mixing and composability of Mamba/GDN exceed the scope of verification. The main text performs channel diagonalization based on Q_c for each frequency, but all experiments use a depthwise core and skip this step. Therefore, the training cost and numerical stability of the complete derivation have not yet been verified. More importantly, if the input depends on the gate's spatial location, multiplying by this gate followed by a Fourier transform generally results in frequency mixing, making it impossible to directly use frequency-wise scalar scans. GDN's rank-one update is also not a typical scalar multiplication.

  5. The compatibility between the CogVideoX fine-tuning target and the inference scheduler needs to be checked first. Appendix E.4 states that the flow-matching target = noise − latents is used, while the inference uses the native CogVideoX DPM setting. The official CogVideoX-5b scheduler is configured as v_prediction [7]; the general diffusion v target is not the same as the linear flow-matching target. Without the corresponding noise path, time parameterization, and output transformation, there may be training/inference mismatch, and baseline LoRA degradation may also be affected. At present, it is not possible to determine whether the program is indeed wrong based on the text alone.

  6. Parameter savings and single-layer timing are insufficient to support broad end-to-end efficiency advantages. Timing primarily uses a fixed C=H=W=32 single-layer fp32 workload. SSM lacks a custom fusion core, therefore it cannot be directly extrapolated to optimize Mamba/GDN or actual high-resolution models. ImageNet achieves higher accuracy by repeatedly inputting T=4 or 8; parameter reduction does not necessarily mean a decrease in computation or latency. The video version retains the original self-attention, modifying the adapter rather than replacing the entire backbone attention; the appendix acknowledges that its training is approximately 2.4 times slower per step.

  7. Pareto Leading Edge and Small Video Improvements Lack Sufficient Independent Statistical Support. Appendice F explicitly lists all single seeds, and G.6 states that the leading edge is the best observation envelope including retries, not average performance. The search range in Table A1 is different, and the T=8/1 setting for S4ND is inconsistent between the text and the table. Figure 5's "Data Efficiency" uses the number of epochs reaching 95% of its peak, without comparing the same accuracy or changing the data volume. Although the video appendix specifies McNemar and bootstrap, the text and figures do not present corresponding p-values ​​and confidence intervals.

  8. Mechanisms of Human-like Circular Reasoning Still Exceed Existing Behavioral Evidence. Models with different values ​​of T are trained separately. The increase in accuracy with T only demonstrates that more computation helps the task, but it does not prove that the same model infers progressively along a contour like humans. With a basic linear core at zero initial state and fixed, repeated input, the final state can be written as Σ_{j=0}^{T−1} A^j x, i.e., a linear filter that changes with T; this is not equivalent to cortical inference with nonlinear state corrections at each step. If the effective core in the frequency domain has been globalized, the explanation of local propagation requires more direct evidence.

Questions for authors
  1. Clearly distinguish between the original learnable core, the discretized transfer core, and the stabilized effective core; plot the inverse Fourier core and the energy/distance distribution outside the support. Compare stabilization that retains finite support, controlled initialization without squash, and direct spatial cycling with the same effective core; also test the cross-edge shortcuts of the cycle boundary. If only approximate local biases can be claimed, they should be limited based on facts, rather than claiming strict equivalence.

  2. Define operators and costs separately for infinite support, zero-filling clipping, and cyclic boundaries, distinguishing between general lower bounds and specific algorithm costs. Correct the multiplication of general blocks, explicitly listing diagonalizability and condition number assumptions. For actual depthwise paths, report the O(TCHW) work of scan, the O(TCHW log(HW)) work of FFT, and projection and memory costs separately; explicitly define O(log T) as the scan span under the fixed-dimensional cost model.

  3. Prioritize comparing the restricted Toeplitz 3×3 core of ConvT3 with the loop design of this paper, which can have a larger core, clearly listing the differences in boundaries, representable operators, stability, and cost. Add a GFNet/AFNO-type frequency-domain filtering and paired loop version to examine whether the benefits exceed existing frequency-domain spatial mixing. See the reviewed references [1] to [4].

  4. Provide complete update formulas for Mamba-sCSSM and GDN-sCSSM, explaining the order of gates, Q/K/V, normalization, FFT, and scan, and defining the conditions for preserving frequency decoupling for each. Separate "integrability into a model" from "maintaining the same equivalence, stability, and parallel complexity." If complete channel mixing is not implemented, the main theorems and claims should be limited to verified subclasses, or the corresponding experimental and feature decomposition costs should be added.

  5. Report the theoretical primitive, JAX single-layer, and practical applications separately. Provide end-to-end throughput, inference latency, peak memory, and training cost for multiple resolutions, channels, sequence lengths, and batches, and compare for the same accuracy and optimization level. Incorporate a shared-weight CNN/Transformer iterative baseline with the same computational budget, a reproducible SHViT training, and LoRA with the same trainable-parameter budget. Do not represent efficiency solely by the number of parameters.

  6. Test different inference steps on the same set of weights, controlling FLOPs, receptive fields, and readout capabilities; compare equivalent filtering, stacked CNNs, and tied-weight Transformers. Use human results with the same stimuli and difficulty to analyze the relationship between accuracy/reaction time and path length and number of interfering factors, and incorporate evidence of intermediate states or message propagation. Limit current conclusions to a tradeoff between accuracy and computational steps for a specific task, rather than establishing a human inference mechanism.

  7. Please compare the following references: [1] J. Hong, Y. Y. Choi, J. Ko, and M. Gwak, “ConvT3: Structured state kernels for convolutional state space models,” ICLR, 2026. [2] M. Mardani, Z. Li, A. Tao, A. Anandkumar, and B. Catanzaro, “Adaptive Fourier neural operators: Efficient token mixers for transformers,” ICLR, 2022.

  8. Section 2, Equation (3), Appendix D.2: The inverse matrix representation of ZOH assumes A is invertible; continuous extensions of zero eigenvalues ​​or expressions using matrix functions should be defined, and it should be stated whether ZOH is actually used in experiments.

  9. Section 3, Page 3: The origin of the finite field dimension (3h)(3w)c and its relationship with the state representation of hwc are unclear and should be supplemented with a derivation; do not directly equate the general representation of finite matrices with cheap diagonalization.

  10. Section 4, Eq. (6): The set membership symbol is missing after C = Circ(K_y); unify the shapes of d, N, c, c_in, and the output channel.

  11. Section 4, Eqs (8) to (14): Explicitly list the existence conditions of Q_c and the treatment of repeated roots/near-defect blocks; general frequency blocks do not guarantee diagonalization of good states. The conjugate symmetry condition of real outputs must also be defined.

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer SwQq

We thank the reviewer for the detailed review. Many points overlap with the other reviews, so we answer the distinct ones here and refer to those responses for the rest.

ConvT3 and Fourier-domain mixers (W3, Q3, Q7). Thank you for the pointer to ConvT3; we were not aware of it and it is directly relevant. ConvT3 obtains 3 x 3 state kernels through a constrained tridiagonal Toeplitz structure, diagonalized by a discrete sine transform. Our construction follows the same principle with a different structure: circulant kernels, diagonalized by the FFT. The differences are that our kernel is unconstrained and of any size (11 x 11 in our experiments, against a 3 x 3 kernel with three free parameters per channel), that we use circular rather than zero-padded boundaries, which is a cost on our side, and that we compose the recurrence with input-dependent SSM circuits (Mamba, GDN) — which DST based transforms can’t obtain without sever expressivity tradeoffs — and evaluate on visual reasoning, ImageNet and video adaptation. The revision cites and discusses ConvT3 and removes our statement that prior convolutional SSMs are confined to 1 x 1 kernels. GFNet, FNO and AFNO are discussed in our response to Reviewer ERiw.

CogVideoX training target (W5). Thank you for flagging this. App. E.4 describes the loss as flow matching, and the reviewer is right that the CogVideoX-5b scheduler is configured for v-prediction. We are checking the training code against the scheduler configuration. The revision will either correct the description in App. E.4 or, if the target was mismatched, remove the experiment. The VideoPhy-2 differences are not statistically significant (see our response to Reviewer uZvP), and no claim in the revised paper depends on them.

Complexity statements (W2, Q2, Q8-Q11). We agree with these corrections. Prop. 1 is the cost of the parallel scan with direct kernel composition, not a lower bound for all spatial ConvSSMs. App. G.3 mixes modulo indexing with growing support and will be rewritten with one boundary convention per case. The constant in Prop. 2 follows Prop. 4 for a general hwc-dimensional system and is not O(hwc). Eq. 26 should be a matrix product, and the elementwise product in Eq. 41 holds only in the diagonal case. We will state the diagonalizability and conditioning assumptions on the per-frequency blocks, report the FFT work O(T c hw log(hw)) separately from the scan, fix Eq. 6, show the (3h)(3w)c dimension, and state that the experiments parameterize the discrete-time transition directly and do not use ZOH.

Remaining points. Locality and stabilization (W1, Q1), channel mixing and gating (W4, Q4), efficiency (W6, Q5), statistics (W7) and the human-like claim (W8, Q6) are answered in our responses to Reviewers ERiw (locality, Fig. 2, ImageNet), uZvP (depthwise transition, gating in the Fourier domain, seeds and Pareto fronts, VideoPhy-2) and dqoY (end-to-end cost). In short, we agree that the stability squash gives the effective kernel full support, that the evaluated model is the depthwise subclass, that gates act per frequency, and that Fig. 4 relates accuracy to computation steps and does not establish a human inference mechanism. The claims are narrowed accordingly. On W7: Table A1 is correct that S4ND sees a single frame on PathFinder, and the sentence in Sec. 5.2 will be fixed; "data efficiency" in Fig. 5 is the number of epochs a run needs to reach 95% of its own peak accuracy, which the caption will state.

Reviewer_ERiwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper proposes spectral convolutional state space models (sCSSMs), a recurrent layer for image sequences in which the hidden state of each frame is updated by a spatial convolution of the previous state. Starting from ConvS5, whose parallel scan has to compose convolution kernels and is therefore restricted to 1×1 state kernels, the paper imposes circular boundary conditions so that the convolutional transition becomes block-circulant and is diagonalised by the 2D DFT.

Critical strengths

1. The core construction is simple. Imposing circular boundaries turns a convolutional state transition into independent per-frequency scalar recurrences. This removes the kernel growth that confined ConvS5 to 1×1 state kernels and makes spatially coupled recurrence compatible with standard associative scans and with gating schemes such as Mamba and GDN. App. E.2 describes the implementation in enough detail to reproduce it, and the code is released. This matters because it offers a cheap way to build spatiotemporally coupled linear recurrences.

2. The visual-reasoning study is controlled. All circuits share the same stem and readout, and the sweep grid is listed in Table A1. Test sets are evaluated once after validation-based early stopping. The NoSpace and NoTime ablations both drop to near chance, while the GDN-sCSSM clearly outperforms ConvS5, S4ND, Mamba-2 and Transformers on both PathFinder-14 and PathTracker. This matters because it is the paper's clearest evidence that spatially coupled recurrence helps on tasks that require incremental grouping.

Critical weaknesses

1. The method is a recurrent form of well-known Fourier global filters, and nothing supports the claim that it realises the brain's local recurrent strategy. Decoupling spatially invariant linear dynamics into per-frequency systems with a spatial Fourier transform is classical [4], and replacing self-attention with learned per-frequency filters on FFT features is the basis of GFNet [1], FNO [2] and AFNO [3], none of which is discussed. The implemented layer is depthwise with identity input and output maps, so the BCCB and QcQ_c derivation of Sec. 4 is unused, and for the repeated static inputs used on PathFinder and ImageNet the plain sCSSM output is exactly (1−K^T)/(1−K^)⊙U^(1-\hat K^T)/(1-\hat K)\odot\hat U, a single GFNet-style global filter. The new element is therefore narrow: a diagonal complex SSM run on each spatial frequency of a frame sequence, with poles given by the DFT of a small kernel. Nor is the recurrence local, as the paper claims, since the squash in Eq. 20 acts per frequency and gives the effective transition full spatial support, and circular boundaries connect opposite image edges. The neuroscience framing also goes beyond its citations: [8, 5, 33] are anatomical studies of horizontal connections in early visual cortex, [33] shows that these differ from the re-entrant feedback that the abstract invokes, [28, 14] support refinement over time but not routing through the same neurons, and the sentence closest to the method, that signals spread locally and build long-range structure over time, is uncited, although incremental-grouping work addresses it directly [5, 6]. In that work response time grows with the distance to be traced, whereas Fig. 4 trains a separate model per TT at one path length, so the claimed human-like trade-off is not tested.

2. The efficiency and accuracy claims are not supported by the reported evidence. The only support for "≈100× the efficiency of the sequential algorithm" is Fig. 2a, which reads about 900 ms against 2.8 ms, roughly 300×, while the text's ≈10× gap between the spatial and sequential scans appears as about 2× in the plot. The sequential baseline also convolves step by step in space, so the gap mixes FFT convolution with parallel scanning. The benchmarks use one 32-channel layer on 32×32 frames, run all SSM baselines through a generic JAX scan, which is why Mamba-2 runs out of memory at T=256T=256, and use a Transformer whose cost grows linearly in TT and is not described. No throughput, FLOPs or memory are given for the ImageNet or video models, although SHViT is speed-oriented and T=8T=8 runs the mixer eight times, and the only end-to-end measurement shows the GDN-sCSSM LoRA to be 2.4× slower per step. The accuracy gains are small and confounded: on ImageNet the baseline is the published SHViT-S1, the paper's own convolutional Hybrid control is already 1.0 point above it, and T=1T=1 without recurrence reaches about 74.6% with T=8T=8 adding only 0.4 points, contrary to the claim that timesteps substitute for depth. On VideoPhy-2 the gain of about 0.9 points is roughly five of 591 prompts from a single seed.

References

[1] Y. Rao, W. Zhao, Z. Zhu, J. Lu, J. Zhou. Global Filter Networks for Image Classification. NeurIPS, 2021.

[2] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, A. Anandkumar. Fourier Neural Operator for Parametric Partial Differential Equations. ICLR, 2021.

[3] J. Guibas, M. Mardani, Z. Li, A. Tao, A. Anandkumar, B. Catanzaro. Efficient Token Mixing for Transformers via Adaptive Fourier Neural Operators. ICLR, 2022.

[4] B. Bamieh, F. Paganini, M. A. Dahleh. Distributed Control of Spatially Invariant Systems. IEEE Transactions on Automatic Control, 47(7):1091–1107, 2002.

[5] A. Pooresmaeili, P. R. Roelfsema. A Growth-Cone Model for the Spread of Object-Based Attention during Contour Grouping. Current Biology, 24(24):2869–2877, 2014.

[6] D. Jeurissen, M. W. Self, P. R. Roelfsema. Serial Grouping of 2D-Image Regions with Object-Based Attention in Humans. eLife, 5:e14320, 2016.

Questions for authors
  • Please report the step times behind Fig. 2 in a table and describe exactly what the sequential baseline computes. How is the ≈100× figure obtained when the plot suggests about 300×? Why does the text say ≈10× for the spatial-domain scan when the plot shows about 2×? Please also add a sequential scan in the spectral domain, so that the speed-up from parallel scanning can be separated from the speed-up from FFT convolution, and specify the Transformer configuration.

  • What are the throughput, FLOPs and peak memory of the GDN-sCSSM SHViT at T∈{1,4,8}T\in\{1,4,8\} compared with SHViT-S1 and SHViT-S2 on the same GPU? What does SHViT-S1 reach when retrained with your recipe?

  • Please check the core weaknesses.

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer ERiw - Part I/II

We thank the reviewer for the close reading and for the references.

Relation to Fourier filtering (GFNet, FNO, AFNO, Bamieh et al.). These are directly related and should have been cited; the revision discusses all four. We do not claim the convolution theorem or per-frequency decoupling as new. The reviewer's reduction is also correct: for the plain sCSSM on a repeated static input, the output at each frequency is u(1 - a^T)/(1 - a), a single global filter. Three things differ from GFNet- and AFNO-style mixers. (1) The filter is the solution of a recurrence over a frame sequence, so with time-varying input (PathTracker, video) it is a spatiotemporal operator, whereas GFNet and AFNO mix within a frame. (2) The poles are the DFT of a k x k kernel, so the layer has k^2 c parameters independent of resolution, where a free per-frequency filter has hwc. (3) The recurrence is an SSM, so input-dependent transitions (Mamba, GDN) apply per frequency. We will state the contribution in these narrower terms.

Depthwise transition and Sec. 4. The reviewer is right that the evaluated layer is the depthwise special case, with an identity input map and a dense output projection. This is stated in App. E.2 and App. F but not in the main text, which is our error. Sec. 4 shows that a channel-mixing transition can be brought to diagonal form by absorbing a per-frequency change of basis Q_c into the input and output maps. We do not evaluate that general model, and the input and output maps of our implementation are not per-frequency, so we do not claim the evaluated model is equivalent to it. All empirical claims are for the depthwise subclass, which is the more constrained model. We will restructure Sec. 4 so that the general BCCB derivation is presented as the general model and the evaluated depthwise instance is stated in the main text.

Locality. The reviewer is right on both counts. Eq. 20 is a contraction that bounds every pole by rho < 1 for stability. Because it acts per frequency, the effective one-step kernel has full spatial support. The same holds for the gating and normalization in the Mamba and GDN variants, which also act in the Fourier domain. In those models the k x k kernel is a parameterization and an inductive bias, not a hard constraint, and circular boundaries do connect opposite edges (Sec. 4.1, App. F). We will say this in the main text and remove wording that implies strictly local recurrence. To test the local case directly, we re-ran Fig. 4 with a vanilla variant that does not include normalisation or gating operations in spectral domain. Its transition is an exact k x k circular convolution, and the revised figure reports that model. (see the results below (*)).

Neuroscience framing. We accept the correction. [8, 5, 33] concern horizontal connections, which are what convolutional recurrence models, and the abstract's "re-entrant feedback" conflates them with inter-areal feedback. We will reword the abstract and introduction to refer to horizontal recurrent connections, and cite Pooresmaeili & Roelfsema (2014) and Jeurissen et al. (2016) for local spreading over time. Fig. 4 trains one model per T at one path length, so it relates accuracy to the number of recurrent steps and does not test the growth of response time with distance. We will drop "human-like" from that claim.

Fig. 2 numbers. We apologize: the ratios quoted in the text were rough estimates, and the reviewer's reading of the plot is more accurate. From the logs at T=1024, the step time is 938 ms for the sequential scan, 1,625 ms for the spatial-domain parallel scan and 2.80 ms for the sCSSM, i.e. 335x and 1.7x. Thank you for the correction. We will replace the estimates with a table of the measured step times. Our main point stands: the spectral scan is more than two orders of magnitude faster than the sequential one at long T, and the spatial-domain parallel scan is slower than both. The reviewer is also right that the sequential baseline convolves in the spatial domain, so the gap combines FFT convolution with parallel scanning; Sec. 5.1 will say so.

Benchmark setup. The SSM baselines use the same generic JAX associative scan as the sCSSM, with no custom kernels for any model (Sec. 5, Setup). The Mamba-2 curve in Fig. 2c is our JAX port run through that shared scan. We also tried the official implementation, and the caption will state which implementation each curve uses and that the OOM reflects the shared scan rather than Mamba-2 with its optimized kernels. The Transformer in Fig. 2c applies multi-head self-attention within each frame, with no attention across time, which is why its cost is linear in T. It is the cheapest Transformer variant, and the caption will describe it. The benchmark is a single 32-channel layer on 32 x 32 frames. App. C states that it does not establish end-to-end throughput, and the only end-to-end timing we have is the 2.4x slower LoRA step (App. F).

AuthorsOpenReview ↗

Rebuttal to Reviewer ERiw - Part II/II

ImageNet. The reviewer's reading is right. Our Hybrid SHViT-S1 control, trained with the same recipe as the sCSSM models, reaches 73.78% and is the fair reference. The GDN-sCSSM reaches 74.62% at T=1, 74.81% at T=4 and 75.03% at T=8 with 5.8M parameters, against the published 72.8% for SHViT-S1 (6.3M) and 75.2% for SHViT-S2 (11.4M). Most of the gain is present at T=1, and going from T=1 to T=8 adds 0.41 points, so we will remove the statement that timesteps substitute for depth and report a small monotonic gain with T. We will add throughput, FLOPs and peak memory for the sCSSM variants against SHViT-S1 and S2 on the same GPU, and limit the claim to parameter efficiency.

VideoPhy-2. The GDN-sCSSM adapter is numerically better than the base model (147 vs 142 Joint successes out of 591 prompts; 24.9% vs 24.0%), but the difference is not statistically significant (paired McNemar test, p = 0.46), and App. F already describes the gain as modest. Each adapter needs about 22 GPU-hours of generation per evaluation, and we did not have the compute to train and score enough independent models for statistical power. We will add the test results to Sec. 5.4 and soften the claims: the abstract and Sec. 5.4 will no longer say that the adapter "beats" the others or "corrects" physical errors.

() results of figure 4 with exact kk kernels :

Timesteps,Seed 42,Seed 43,Seed 44,Mean 1,55.835,55.465,56.000,55.767 2,58.270,58.185,57.305,57.920 3,61.210,61.310,61.215,61.245 4,63.815,63.170,62.960,63.315 5,65.915,68.250,66.470,66.878 6,71.155,71.290,71.780,71.408 7,75.655,75.730,73.800,75.062 8,78.795,78.080,77.770,78.215 9,81.310,81.805,81.835,81.650 10,83.915,83.670,83.540,83.708 11,83.980,84.845,84.560,84.462 12,84.030,84.255,84.800,84.362 13,85.145,83.790,84.235,84.390

I thank the authors for the candid response. It resolves the framing and presentation issues I raised. Therefore, I update my presentation score. However, the two critical weaknesses largely remain, and I keep my overall score.

On novelty, the response accepts that the plain sCSSM on static input is a single global filter and narrows the contribution to three points. The second point does not distinguish the method from FNO and AFNO, whose parameters are also independent of resolution. What remains new is a linear, scan-parallel recurrence per frequency with poles parameterized by a small kernel, which is narrow relative to the claims the paper makes.

For efficiency and accuracy, the corrected step times confirm that the 335× gap combines FFT convolution with parallel scanning, but the sequential spectral scan that would separate the two is not provided. The response also says that the official Mamba-2 implementation was tried without reporting the result, and that the Transformer in Fig. 2c attends only within each frame, so the ≈14× speed-up is measured against a model that does not mix information across time. Throughput, FLOPs and memory for the ImageNet are again promised rather than reported. The ImageNet numbers now show gains of 0.84 points at T=1 and 1.25 points at T=8 over the fair Hybrid control. With the zero initial state implied by the scan, the transition never acts at T=1, so about two-thirds of this gain does not come from the proposed spatial recurrence, which itself contributes 0.41 points. On VideoPhy-2, the McNemar test with p = 0.46 confirms that there is no measurable gain.

Reviewer_uZvPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the spectral convolutional state space model (sCSSM), which lifts convolutional recurrence into the Fourier domain. Under circular boundary conditions the convolution is diagonalized by the 2-D DFT, so each frequency evolves as an independent scalar recurrence. A parallel scan then runs in O(T)O(T) work and O(log⁡T)O(\log T) depth without growing the kernels. The authors compose sCSSM with Mamba and Gated Delta Networks, and report results on PathFinder/PathTracker, ImageNet-1K (replacing attention in SHViT) and VideoPhy-2 (a LoRA adapter on CogVideoX-5B).

Critical strengths

Under circular boundaries, the convolutional recurrence is diagonalized by the 2-D DFT, so scan composition uses fixed-size spectral coefficients and the kernel no longer grows with the number of steps (App. E.2 makes the per-frequency scalar recurrence and its associativity explicit). This is a sound, simple idea that could be useful as a building block for efficient spatiotemporal models.

Anonymized code is provided, the compute budget (including discarded runs) is disclosed, and the appendix openly states limitations (depthwise-only, single seed, modest VideoPhy-2 gain, slower training). The Hybrid-SHViT control and the NoSpace/NoTime ablations are the right kind of controls, although they are not sufficient.

Critical weaknesses

All results are single-seed. The "new Pareto front" on PathFinder/PathTracker (Fig. 5) is a best-observed envelope over configurations (App. G.6) with very unequal grid sizes per architecture (Table A1), and it is unclear whether the front was selected on validation or test accuracy. The natural baselines are missing: nonlinear convolutional recurrences (ConvLSTM/ConvGRU/hGRU, which motivated these benchmarks), large-kernel CNNs, and FFT-based global filters or a 3-D FFT convolution. With static inputs repeated TT times (PathFinder, ImageNet), the plain sCSSM reduces to a fixed large-receptive-field linear filter, so "recurrence substitutes for depth" and the "human-like time–accuracy trade-off" (a separate model trained per TT) are not demonstrated. The ImageNet gain is <2<2 points, single-run, reported only by parameter count (no FLOPs/throughput) and without compute-matched controls. The VideoPhy-2 gain is about 1 pp on 591 prompts, with no p-values or confidence intervals reported despite stated tests (App. E.4). This is consequential because these results are the paper's only evidence that the method is useful.

The paper's motivation is GPU parallelism, with O(log⁡T)O(\log T) depth and "no compromises on what matters" (Sec. 4.1), yet the evaluated implementation is depthwise only, with channel mixing explicitly untested (App. E.2, F). It also uses no ZOH discretization. The experiments use T≤32T \le 32, where sequential recurrence is cheap, and the speed advantage is shown only on a single-layer random-data benchmark against full attention over ~10610^6 tokens. Composition with Mamba/GDN is claimed to be seamless, but FFT diagonalization requires a spatially translation-invariant transition. The paper does not explain how position-dependent gates or the delta rule are handled, and the GDN variant (used in all downstream experiments) is about 12× slower than the plain version for which the efficiency numbers are reported. Neuroscience claims (Sec. 6) rest on a linear, circular, depthwise model with no neural or behavioural data. This matters because the contribution is framed as a general, scalable primitive, which the evidence does not establish.

Questions for authors
  1. Which exact model produced the "single sCSSM layer approaches human accuracy" result (plain or GDN, depth, TT)? Please report mean ± std over at least 3 seeds, with configurations selected on validation accuracy only.
  2. How is GDN/Mamba gating made compatible with FFT diagonalization, which requires a spatially translation-invariant transition? Are the gates spatially uniform, and how is the delta-rule update computed in the frequency domain? Please give the full equations for GDN-sCSSM.
  3. In the static-input settings (PathFinder, ImageNet), how does the plain sCSSM differ from a single depthwise large-kernel convolution (or its closed-form geometric-series equivalent) at matched parameters and receptive field?
  4. What exactly was run as "ConvS5" with k>1k>1 (Sec. 4.1 says it is restricted to 1×11\times1, but Table A1 and the Fig. 2 caption use larger kernels)? Is there any measurement of the memory blow-up or training instability shown in Fig. 1?
2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer uZvP - Part I/II

We thank the reviewer for a careful and fair review, and for noting that the appendix states the main limitations openly. Several of the points below were disclosed only in the appendix. We agree they belong in the main text and will move them there.

Single seed and the Pareto fronts. All results are single-seed (App. F), and the fronts in Fig. 5 are best-observed envelopes over the sweep, not expected performance (App. G.6). The visual-reasoning sweeps alone are about 297 training runs, and we do not have the compute to replicate them over seeds. We will make the evidence easier to judge without new training: Fig. 5 will show every configuration of every architecture, a new table will give the median and range of accuracy across each architecture's grid, and "new Pareto front" becomes "best trade-off observed in our sweep". Checkpoints are selected by early stopping on the validation split, and the plotted value is the test accuracy of that checkpoint (Sec. 5.2); the envelope is drawn over those values. On grid sizes (Table A1): the largest grid belongs to a baseline (Transformer, 32 configurations), and the plain sCSSM has 8, the same as ConvS5, GDN and Mamba-2. GDN-sCSSM does have a larger grid (24) than GDN (8), which the per-architecture table will make visible.

Baselines. Nonlinear convolutional RNNs (ConvLSTM, hGRU) motivated these benchmarks. App. B explains why they are not our comparison class: they are inherently sequential, and our question is whether a parallelizable model can reach their regime. We agree the reader needs the reference point and will add their published results on these benchmarks [20, 22], noting where the stimulus settings differ. FFT-based global filters (GFNet, AFNO) are discussed in our response to Reviewer ERiw.

Static inputs. The reviewer is right. For the plain sCSSM with a repeated static input, the state after T steps at each frequency is u(1 - a^T)/(1 - a): one linear spectral filter whose shape depends on T. What T adds is reach. With an exact k x k transition, the T-step operator is a polynomial of degree T-1 in the kernel with support T(k-1)+1, so T steps give a weight-tied filter that grows from local to global with k^2 c parameters. The reduction does not hold for PathTracker and video, where the input changes over time, or for the Mamba and GDN variants, whose transitions depend on the input. We will remove "recurrent timesteps substitute for depth" from Sec. 5.3 and present PathFinder and ImageNet as validation on static inputs, with PathTracker and VideoPhy-2 as the spatiotemporal tests.

ImageNet. The fair reference is our Hybrid SHViT-S1 control, trained with the same recipe (73.78%). The GDN-sCSSM reaches 74.62% at T=1 and 75.03% at T=8 with 5.8M parameters, against the published 72.8% for SHViT-S1 (6.3M) and 75.2% for SHViT-S2 (11.4M). The gain is 0.84 points over the control at T=1 and a further 0.41 points from T=1 to T=8. We will report FLOPs, throughput and peak memory for all variants and limit the claim to parameter efficiency.

VideoPhy-2. The GDN-sCSSM adapter is numerically better than the base model (147 vs 142 Joint successes out of 591 prompts; 24.9% vs 24.0%), but the difference is not statistically significant (paired McNemar test, p = 0.46), and App. F already describes the gain as modest. Each adapter needs about 22 GPU-hours of generation per evaluation, and we did not have the compute to train and score enough independent models for statistical power. We will add the test results to Sec. 5.4 and soften the claims: the abstract and Sec. 5.4 will no longer say that the adapter "beats" the others or "corrects" physical errors.

Depthwise transition. All experiments use a depthwise (per-channel) transition. This is stated in App. E.2 and App. F but not in the main text, which is our error; Sec. 5 will state it and the "no compromises" sentence in Sec. 4.1 will be reworded. Sec. 4 shows that a channel-mixing transition can be brought to diagonal form by absorbing a per-frequency change of basis Q_c into the input and output maps. We do not evaluate that general model, and the input and output maps of our implementation are not per-frequency, so we do not claim the evaluated model is equivalent to it. What we evaluate is the depthwise subclass, with channel mixing done outside the recurrence (stem, output projection, and the Q/K/V projections of the GDN variant). All empirical claims are for that subclass, which is the more constrained model. The general form would add a c x c eigendecomposition per frequency, O(hw c^3), and requires each block to be diagonalizable and well conditioned; we will state these costs and assumptions.

AuthorsOpenReview ↗

Rebuttal to Reviewer uZvP - Part II/II

Time-accuracy trade-off. Fig. 4 trains one model per T at a single path length, so it relates accuracy to the number of recurrent steps. It does not test the human finding that response time grows with path length, and we will drop "human-like" from that claim. The revised Fig. 4 uses a strictly local variant of the sCSSM (see our response to Reviewer ERiw).

Discretization, sequence length and speed. The implementation parameterizes the discrete-time transition directly (Eq. 19) and does not use ZOH. The ZOH in Sec. 2 is background, and we will say so. Our tasks use T <= 32. On the single-layer benchmark the logged gap to a sequential scan is 22x at T=8 and 74x at T=32 (335x at T=1024), so the scan already helps at the lengths we train on, but this is a single-layer measurement. App. C states that it does not establish end-to-end throughput, and we will move that statement to Sec. 5.1. The 12x cost of the GDN variant is reported in Sec. 5.1, and the efficiency axes of Fig. 5 use each variant's own measured step time, so the GDN-sCSSM points already include it.

Composition with Mamba and GDN (also Q2). The reviewer is right that a position-dependent gate applied to the state in pixel space would couple frequencies and break the diagonalization. We do not do that. In GDN-sCSSM, queries, keys and values are projected from each frame and then transformed with the FFT, and the keys are L2-normalized. At every spatial frequency the state S_t is a small d_k x d_k complex matrix (d_k = 2 by default), updated as S_t = M_t S_{t-1} + beta_t v_t k_t^H with M_t = alpha_t W (I - beta_t k_t k_t^H). Here W is the squashed DFT of the learned k x k kernel, and alpha_t and beta_t are decay gates computed from frame t. The readout is o_t = S_t q_t, followed by the inverse FFT, RMSNorm and a SiLU output gate. Each step is an affine map on a fixed-size matrix, so the maps compose associatively and the same scan applies per frequency with a carry that does not grow. In the Mamba variant the input-dependent gates are likewise applied per frequency after the FFT. The consequence is that gating is frequency-selective rather than position-selective, which differs from Mamba and GDN on pixel sequences. We will state this in Sec. 4.2, replace "seamlessly", and add the full update equations for both variants as an appendix.

Neuroscience claims. We collect no neural or behavioral data. Human accuracy is taken from [15, 22] (Sec. 5.2), and App. F notes that linear dynamics and circulant boundaries limit the model as an account of cortical circuits. We will reword the Discussion from "direct implications" to a statement that the model class makes such comparisons tractable.

Q1. The claim refers to depth-1 models: a stem, one sCSSM block and a readout. The human reference values are 89% on PathFinder-14 [15] and 90% on PathTracker [22]. The revision will state the exact configuration (variant, embedding dimension, T) and its accuracy next to the claim, and will keep the phrase "single layer" only for depth-1 results. We cannot provide three seeds; see above for what we report instead.

Q3. In the static setting the plain sCSSM is a spectral filter (see "Static inputs"). Compared with a single depthwise large-kernel convolution, an exact k x k transition reaches a receptive field of width T(k-1)+1 with k^2 parameters per channel: 81 x 81 from 121 parameters at k=11 and T=8, where a free kernel of that reach needs 6,561. We do not have a matched-receptive-field convolutional control, and our NoTime ablation is not one, because with a zero initial state the kernel is not applied at T=1. We will state this as a limitation.

Q4. ConvS5 was run with 1 x 1 state kernels, as in [31]. The Fig. 2 caption and Table A1 are wrong on this and will be corrected; thank you for catching it. The "CSSM (no FFT)" curve in Fig. 2a is the k=11 spatial-domain parallel scan, which shows its time cost. Fig. 1 claimed memory explosion and unstable training for ConvS5 with K>1, neither of which we measured. We have reworded that row of Fig. 1 to "kernel grows with T", which is what Prop. 1 and App. E.2 establish.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.