acceptodds
Withdrawn from ICLR 2027

ViS: Vision State-Space Token Mixer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

State-space models (SSMs) bring linear-complexity sequence modeling to vision, but only after the image is flattened into a 1D sequence under a handcrafted scan order, a causal bias that multi-directional scanning mitigates rather than removes. In contrast, we view SSMs from the perspective of token mixing: each token can aggregate information from other tokens by iteratively updating its own hidden state, as in an SSM recurrence. This perspective suggests that what must be defined is not a single global scan order, but a principled order in which each token aggregates its context. We introduce the Vision State-Space Token Mixer (ViS), which redefines the scan axis as spatial distance: every token's state absorbs information from concentric rings around it, yielding a scan-order-free token mixer with a global receptive field. Despite the apparent difficulty of parallelizing such position-wise recurrences, we derive an exact closed-form formulation as structured masked attention, enabling efficient implementation on modern attention-friendly hardware. Experiments across image classification, segmentation, object detection, and generation demonstrate ViS as a viable general-purpose vision backbone; ViS achieves up to higher throughput while maintaining comparable or better performance, and exhibits the strongest zero-shot resolution transfer among vision backbones through its coordinate-computed spatial prior.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
State-Space Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_i2SsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Vision State-Space Token Mixer (ViS), a state-space operator retaining 2D data format. Instead of flattening an image into a 1D sequence with a handcrafted scan order, ViS lets every token run its own recurrence over spatial Chebyshev-distance rings around it, inputting rings from farthest to nearest. This position-wise recurrence, under a scalar value parameterization of the matrix A in SSM, admits an exact closed form as structured masked attention within the SSD framework (Mamba-2), enabling GEMM-friendly parallel computation. To restore directional information lost by the isotropic formulation of the ring, the authors add a sector gain on the mask and a rotary embedding on the B/C projections. Experiments on classification (ImageNet-1k), detection (COCO), semantic segmentation (ADE20K), and class-conditional image generation (ImageNet-1k) show superior accuracy and zero-shot resolution transfer.

Critical strengths
  • Novelty and soundness of the spatial state space model: The reframing of SSMs as per-token context aggregation over spatial distance rather than sequence position after flattening patches is conceptually novel, and the reviewer agree with the merit.
  • zero-shot resolution robustness: The injected spatial prior (direction rather than absolute grid position) is well-motivated to be robust to the resolution difference and actually gives the best accuracy at every tested resolution up to 768², with a widening margin.
Critical weaknesses
  • Claims regarding inference speed might be overstated: Although ViS computes point-wise recurrences, it claims to achieve high speed through GEMM operations. However, the primary baselines are Mamba-based methods that especially combine 1D scans with 2D convolutions, which incur overhead from data format conversions. The reviewer believes the paper should also discuss methods such as MambaVision [1], SF-Mamba [2], and Adventurer [3], which do not use 2D convolutions and can better leverage the computational efficiency of Mamba itself. Furthermore, including the convolution-based MambaOut in Fig. 6 seems to suggest that ViS loses to MambaOut in terms of the throughput-performance trade-off. While ViS is parameter-efficient, is it also throughput-efficient?
  • Insufficient technical details: The paper states that it adopts a Mamba-2-style computation, but several implementation details remain unclear. For example, does it use multiple heads for the hidden state at each point, and what is the head dimension? Could the point-wise hidden state be achieved while reducing computational costs by adjusting these parameters?

[1] MambaVision: A Hybrid Mamba-Transformer Vision Backbone, CVPR2025
[2] SF-Mamba: Rethinking State Space Model for Vision, ICML2026
[3] Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiency, CVPR2026

Questions for authors
  • Details of the experimental setup: Could the authors provide more details about the state space model configurations used in the experiments?
  • Ablation study on computational cost: Given the use of point-wise hidden states and a quadratic-complexity computation scheme, ViS appears to lose the advantages that state space models and linear attention typically have over full attention. I would be interested in seeing how inference speed and FLOPs scale with increasing image resolution under both the quadratic and recurrent formulations.
  • Robustness to resolution changes: Previous studies have shown that incorporating RoPE can improve robustness to changes in resolution [4]. Comparing ViS with methods that incorporate RoPE, such as RoPE-ViT [4] and MILA [5], could help establish whether the proposed 2D state space model offers additional benefits beyond those attributable to RoPE alone.

[4] Rotary Position Embedding for Vision Transformer, ECCV2024.
[5] Demystify Mamba in Vision: A Linear Attention Perspective, NeurIPS2024.

Reviewer_778SOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the scan-order bias in 2D Vision State-Space Models (VSSMs) by proposing the Vision State-Space Token Mixer (ViS). ViS redefines the scan axis as spatial distance: every token’s state absorbs information from concentric rings around it, yielding a scan-order-free token mixer with a global receptive field. ViS instantiates an independent hidden state at every spatial position and updates it sequentially by aggregating tokens over concentric Chebyshev rings from the farthest ring to the nearest. Empirical evaluations are conducted across ImageNet-1K, COCO, ADE20K datasets.

Critical strengths
  1. Reframing 2D spatial context accumulation around per-token Chebyshev distance rings is clean and motivated.
  2. The 2D prefix-sum contraction is a good observation.
Critical weaknesses
  1. The method already exists under other names, and none of them are cited. Eq. 6–7 is RMT’s Manhattan Self-Attention (CVPR’24)[1] with softmax removed, Manhattan was substituted to Chebyshev. The comparation between RMT and ViS both reported from paper:

At RMT vanilla recipe:

RMT-S 27M/4.5G/ 84.1 vs ViS-T 28M/5.3G/84.1;

RMT-B 54M/9.7G/85.0 vs ViS-S 53M/9.1G/84.7;

RMT-L 95M/18.2G/85.5 vs ViS-B 90M/16.6G/85.4.

A softmax-attention model from 2023 matches or beats ViS at every scale with a weaker recipe, not to mention the results of RMT's other improvements but stronger performance. Importantly, ViS heavily relies on MESA self-distillation and heavy regularization.

RMT best result reported from paper:

RMT-S 27M/4.5G/84.8 vs ViS-T 28M/5.3G/84.1;

RMT-B 54M/9.7G/85.6 vs ViS-S 53M/9.1G/84.7;

RMT-L 95M/18.2G/86.1 vs ViS-B 90M/16.6G/85.4.

The introduction of SSM to vision does not demonstrate any practical significance.

  1. Confounded Comparisons & Baseline Degradation: ViS heavily relies on MESA self-distillation and heavy regularization, which are absent in baseline models. When compared fairly against DAMamba, ViS-S uses 17.8% more parameters yet achieves lower Top-1 accuracy (84.7% vs. 84.8%), and ViS-T underperforms DAMamba-T on ADE20K segmentation. Furthermore, DAMamba inexplicably drops by 0.5% in the controlled ablation (Table 7c), indicating over-optimized macro-parameters favoring ViS. And with the correct baselines in the table, ViS is not SOTA.

  2. The sub-quadratic claim never cashes out, and the paper’s own Tab. 5 shows why. At 224² the sub-quadratic form is cheaper only in stages 1–2 and more expensive in stages 3–4; measured, it is 14× slower everywhere.

  3. The proof of Corollary 3.1 in Appendix B.2 strictly relies on isotropic ring weights and axis-aligned rectangular Chebyshev balls to enable O(N)O(N) ring-sum reductions via a 2D prefix-sum table (integral image) with 4 lookups. However, introducing 8-direction sector gains in Sec. 3.4 breaks ring weight uniformity and partitions the grid into diagonal-bounded triangular wedges rather than axis-aligned rectangles. Consequently, the 4-lookup rectangular prefix-sum algorithm breaks down mathematically under Sector Gain, creating an irreconcilable theoretical contradiction with Appendix B.2.

  4. Tab. 7(b) reverses the ring order, which inverts the prior so that the farthest tokens are weighted most and the token itself least. This leads the drop from 84.1 to 83.6. the same drop number 0.5 as removing eight sector scalars. Either the model learns to flatten the decay, or the radial prior is worth half a point; both readings undercut the paper.

  5. The primary motivation for adopting vision SSMs over Transformers is to escape quadratic computational complexity. However, Table 5 and Sec. 4.2 reveal that the proposed sub-quadratic recurrent form is 14 to 24 times slower than the quadratic matrix form on an H100 GPU, forcing the authors to use the O((HW)2)O((HW)^2) dense matrix multiplication form across all experiments. Once executed in this quadratic closed form with positional rotations, ViS mathematically degenerates into standard unnormalized quadratic Linear Attention equipped with an ALiBi-style exponential distance decay mask. Branding a dense quadratic attention operator as a 2D SSM merely because exponential decay allows a trivial scalar recurrence unrolling is a severe conceptual over-claiming. Furthermore, Table 5 only benchmarks a tiny 56×5656 \times 56 token grid, completely hiding the quadratic memory and compute bottlenecks under high-resolution dense prediction inputs.

  6. Missing to compare and illustrate the difference with other adaptive sampling vision ssm work beyond DAMamba, such as DefMamba: Deformable Visual State Space Model [CVPR 2025] and Deformba: Vision State Space Model with Adaptive State Fusion [ICML 2026].

Overall, I think this paper concurrently packaged as an SSM while running as quadratic attention, with broken prefix-sum proofs under sector gains and underperforming matched baselines.

[1]: RMT: Retentive Networks Meet Vision Transformers

Questions for authors
  1. How can the 4-lookup rectangular 2D prefix-sum proof in Appendix B.2 remain valid after introducing diagonal-bounded 8-direction sector gains in Sec. 3.4?
  2. What are the actual wall-clock execution time and peak GPU memory of the quadratic closed form vs. the recurrent form when scaling token grids from the tiny 56256^2 benchmark to higher-resolution Stage-1 feature maps?
  3. Does any ViS variant remain on the Pareto front if you add the another similar work's results such as RMT?
Reviewer_dNgdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose a token mixer in which every token keeps its own hidden state and absorbs the mean of each Chebyshev ring around it, from the farthest ring to the nearest, namely ViS. Experiments show SOTA, or, on-par accuracy and up to 5.1× higher throughput than prior vision SSMs.

Critical strengths
  • In what order should each token aggregate its context do deserve an investigation.

  • The evaluation is comprehensive.

  • The paper is well-organized and clearly illustrated.

Critical weaknesses
  • RMT have a manhattan self-attention that applies a spatial decay matrix based on Manhattan distance to attention, extending RetNet's temporal decay to 2D. ViS differs mainly in using Chebyshev rather than Manhattan distance, an input-dependent rather than per-head decay, no softmax, ring-size normalization, and the sector gain/RoPE add-ons. The author should add comparison and discussion with this method.

  • Even claimed to be linear complexity, Table 5 shows the recurrent form is about 14× slower, so every result uses the O((HW)^2 N) fused kernel. In practice ViS is an attention operator. The abstract's framing around the linear complexity of SSMs, and claims of "retaining the benefits of an image-native SSM", are therefore misleading.

  • The training recipe is also not fair. ViS uses MESA self-distillation from an EMA teacher. However, most of the baselines are without MESA, say, Swin, ConvNeXt, Vim, VMamba, LocalVMamba, MSVMamba. In my experience, this could raise the performance by 1 or 2 pp in imagenet-1k classification. Please also report w/o MESA performance at the same time.

Questions for authors

Please answer the question I posed in the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.