- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes spectral convolutional state space models (sCSSMs) for parallelizing spatially coupled recurrence. The main idea is to represent circulant convolution operators in the Fourier domain, where recurrent kernel composition becomes pointwise multiplication. The authors derive the corresponding SSM formulation and show that it supports parallel scans. Experiments include runtime benchmarks, PathFinder and PathTracker, ImageNet classification, and LoRA adaptation of a video generation model.
Critical strengths
The paper addresses a concrete limitation of convolutional SSMs: spatially coupled recurrence is expensive to parallelize. The spectral formulation provides a reasonable solution under circulant boundary conditions, and the runtime experiments show clear improvements over the spatial implementation. The controlled visual reasoning experiments are also useful, particularly the ablations removing spatial or temporal recurrence.
Critical weaknesses
Section 4 develops a general spectral SSM with channel mixing within each frequency block. However, Appendix F states that the actual implementation uses depthwise spatial kernels, reducing these blocks to scalar transitions. The more general formulation, including the proposed change of basis for channel mixing, is therefore not evaluated. This makes it difficult to assess how much of the theoretical framework is practically useful beyond the depthwise case.
The strongest results are obtained on PathFinder and PathTracker, where spatially local recurrence is particularly well suited to the task. On ImageNet, the comparison is limited to a small SHViT backbone, and increasing the number of recurrent steps also increases computation. The paper mainly reports accuracy against parameter count rather than a controlled accuracy-latency comparison.
The video generation results are less convincing. Appendix F reports only a 1–2 percentage point improvement on VideoPhy-2 over the original CogVideoX model, while training is approximately 2.4 times slower than the other LoRA variants. All reported results use a single random seed. Without uncertainty estimates, it is difficult to judge whether this relatively small improvement is reliable.
The theoretical result concerns the work and parallel depth of the recurrent scan, not the complete model. FFT operations, complex-valued intermediate states, and additional transformations also contribute to runtime and memory usage. Although the single-layer benchmarks are useful, they do not establish that the proposed method is more efficient in end-to-end vision applications. The video adaptation experiment, in fact, shows a substantial training-time overhead.
Questions for authors
-
Why is the full channel-mixing formulation not evaluated? Does it provide any benefit over the depthwise implementation, and how does its computational cost compare?
-
Can the authors provide ImageNet results under matched inference latency or FLOPs, rather than only matched parameter counts?
-
Is the 1–2 percentage point improvement on VideoPhy-2 consistent across random seeds? How does the method compare when the additional training time is taken into account?
-
How much of the reported runtime advantage remains when FFT overhead and the complete model are included?
Rebuttal to Reviewer dqoY
We thank the reviewer for the clear and fair assessment. The weaknesses listed are limitations we report in App. F, as the reviewer notes, and we agree they should carry more weight in the main text.
W1 / Q1: full channel mixing. All experiments use a depthwise (per-channel) transition. This is stated in App. E.2 and App. F but not in the main text, which is our error; Sec. 5 will state it and the "no compromises" sentence in Sec. 4.1 will be reworded. Sec. 4 shows that a channel-mixing transition can be brought to diagonal form by absorbing a per-frequency change of basis Q_c into the input and output maps. We do not evaluate that general model, and the input and output maps of our implementation are not per-frequency, so we do not claim the evaluated model is equivalent to it. What we evaluate is the depthwise subclass, with channel mixing done outside the recurrence (stem, output projection, and the Q/K/V projections of the GDN variant). All empirical claims are for that subclass, which is the more constrained model. The general form would add a c x c eigendecomposition per frequency, O(hw c^3), and requires each block to be diagonalizable and well conditioned; we will state these costs and assumptions. We have not measured whether channel mixing in the recurrence helps.
W2 / Q2: ImageNet under matched compute. We do not have matched-latency results. Repeating the input T times runs the sCSSM mixer on T frames, so the T=8 model costs more than SHViT-S1 at inference, and parameter count is not a proxy for compute. We will report FLOPs, throughput and peak memory for every variant and limit the claim to parameter efficiency. Relative to our Hybrid SHViT-S1 control, which is trained with the same recipe (73.78%), the GDN-sCSSM gains 0.84 points at T=1 (74.62%) and a further 0.41 points at T=8 (75.03%).
W3 / Q3: VideoPhy-2. The GDN-sCSSM adapter is numerically better than the base model (147 vs 142 Joint successes out of 591 prompts; 24.9% vs 24.0%), but the difference is not statistically significant (paired McNemar test, p = 0.46), and App. F already describes the gain as modest. We did not run multiple seeds: App. E.4 reports a single seed, each adapter needs about 22 GPU-hours of generation per evaluation, and we did not have the compute to train and score enough independent models for statistical power. We will add the test results to Sec. 5.4 and soften the claims: the abstract and Sec. 5.4 will no longer say that the adapter "beats" the others or "corrects" physical errors. The sCSSM adapter is 2.4x slower per training step (App. F), and we make no claim at matched training time.
W4 / Q4: end-to-end cost. The step times in Fig. 2 are measured forward plus backward and include the FFT and inverse FFT. Prop. 3 covers only the scan, and we will add the rest: the FFTs add O(T c hw log(hw)) work with depth independent of T, and complex states double per-block memory (App. F). The reviewer is right that a single-layer benchmark does not establish end-to-end efficiency. App. C says this, and the only end-to-end timing we have, the LoRA, is slower. The revision moves both statements into Sec. 5.1.