acceptodds
Under review as a conference paper at ICLR 2027

VTDS: A Vision Semantic Time-Delay State Space Mixer for Efficient Long-Range Visual Modeling

Abstract

Visual state space models (SSMs) provide an efficient alternative to self-attention for long-range visual representation learning. However, existing approaches mainly improve state transition dynamics or spatial scan paths, while the content-dependent semantic distance between a token and its relevant context remains implicit. To address this limitation, we propose VTDS, a Visual Semantic Time-Delay State Space Architecture, which explicitly incorporates token-wise semantic delay modeling into visual state propagation. VTDS learns a differentiable dependency distribution over a bounded historical window to retrieve semantically relevant context, and adaptively integrates the retrieved information with the recurrent state through a bounded fuzzy regulator. A loop-free diagonal recurrence further enables parallel state propagation. For a fixed lag window, VTDS scales linearly with sequence length without constructing a global token-wise attention matrix. Experiments on ImageNet-1K classification, COCO object detection and instance segmentation, and ADE20K semantic segmentation demonstrate competitive accuracy-efficiency trade-offs across diverse visual recognition tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual State Space Architecture, Semantic Lag Estimation, Fuzzy Semantic Regulator

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tDZwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper observes the mismatch in sequential distance to semantic relevance, which occurs when a 2D image is serialized into a 1D sequence. The paper tackles this mismatch with the proposed VTDS framework, which works by extracting the relevant information within a bounded window to then fusing it with the information gathered through a simplified diagonal recurrence using a fuzzy semantic regulator with enhanced spatial coverage via alternating spatial scans across layers. The experiments show that VTDS achieves competitive performance across datasets and tasks, although with a noticeably higher compute cost than existing visual SSMs.

Critical strengths
  1. The paper addresses a relevant problem in the field, where converting a 2D image into a 1D sequence with a fixed serialization order leads to a clear mismatch between the spatial and semantic relationships. The semantic retrieval mechanism proposed to address this problem is an interesting approach which seems to complement well with the state propagation recurrence and with the added learnable retrieval distribution seems to provide with some interpretability of the model's behavior.
  2. A clear ablation study has been performed to justify the design choice, as it shows a clear incremental increase in performance with the addition of all the proposed components, especially with the largest improvement coming from the semantic lag estimator.
  3. The model evaluated across various vision tasks suggests some potential benefits for the proposed framework on dense prediction tasks, despite the larger compute budget.
Critical weaknesses
  1. While the core motivation of the paper being long-range semantic dependency modeling, the proposed framework does not seem to sufficiently support it. The recurrence is modeled to add semantic relations with the use of a semantic lag estimator, but with a very limited fixed history (L=10), it covers only a very limited portion, especially when considering larger images. While the overall information coverage could be considered to be longer with the use of alternating directions and recurrent propagation, the proposed semantic retrieval remains local. This raises a question of whether the improvements come from local-content-based aggregation or long-range semantic modeling.

  2. Architectural Novelty and trade-offs are not sufficiently justified with the use of existing approaches. Works like LocalMamba (Huang et al., 2024) have already explored the serialization and scan-direction limitation, while many other works like Mamba-Adaptor (Xie et al., 2025) and TV3S (Hesham et al., 2025) address the serialization problem for long-range modeling. Furthermore, the use of a semantic lag estimator for close-range content assessment seems to align with local content-based attention that has been explored in methods like Neighbourhood Attention (Hassani et al., 2023). VTDS indeed simplifies Mamba's content-selectiveness with the use of fixed diagonal recurrence to allow for parallelization, but adds a retrieval and fuzzy attention mechanism that increases the compute. It is not clear in the current state whether the advantage coming from the added complexity is meaningful over a straightforward use of local attention + recurrence or maybe just the existing selective SSM.

  3. The accuracy-efficient claim in the paper is not convincingly demonstrated in the experimental results. Table 1 clearly shows the higher requirement of VTDS to achieve similar performance to Mamba-T, and is similarly seen in the other results. While Table 4 shows some favorable throughput, its FLOP count seems to differ from the Table1, the batch size and throughput seems inconsistent. This makes the claims in the paper difficult to assess.

Questions for authors
  1. With the proposed retrieval having a limited and fixed historical window, how can the authors demonstrate that the reported improvement comes from the semantic relations rather than local feature aggregation? And how can the mechanism remain effective as the image resolution, and hence the sequence lengths, increases beyond the limited retrieval window (L=10)?

  2. What is the fundamental advantage of the proposed methods, which use semantic retrieval in local windows, over existing selective SSMs or a simpler local-attention+recurrence architecture? A controlled comparison would really help clarify the novelty of the proposed design.

  3. How do the authors justify the claim made in the paper on the accuracy-efficiency trade-offs when the proposed methods have substantially higher compute and parameters than the current SSMs? Could the authors clarify the inconsistencies in the results in Tables 1 and 4?

Reviewer_jMDqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes VTDS, a visual state space model that explicitly captures semantic dependencies between visual tokens through differentiable semantic lag estimation, fuzzy semantic regulation, and parallel diagonal recurrence. Experiments on ImageNet-1K, COCO, and ADE20K demonstrate its applicability to various visual recognition tasks. While the motivation is reasonable, concerns regarding methodological novelty, computational efficiency, and experimental competitiveness limit the overall contribution.

Critical strengths
  1. The paper identifies a meaningful limitation of visual SSMs regarding the mismatch between sequential distance and semantic relevance.
  2. The proposed framework integrates semantic retrieval and recurrent state propagation in a coherent architecture.
  3. Experiments across multiple benchmarks and component-wise ablations provide evidence of the method's effectiveness.
Critical weaknesses
  1. Limited methodological novelty. The semantic lag estimator essentially performs window-based similarity-weighted feature aggregation, while the fuzzy regulator resembles conventional adaptive gating. The technical contributions appear largely incremental.
  2. Unconvincing accuracy-efficiency trade-off. VTDS requires 70.0G FLOPs to achieve 82.5% ImageNet Top-1 accuracy, whereas VMamba-T achieves 82.6% with only 4.9G FLOPs. This substantial computational overhead contradicts the paper's emphasis on efficiency.
  3. Insufficient evidence for long-range modeling. The semantic retrieval window is restricted to only 10 historical tokens. The paper lacks convincing experiments demonstrating that this mechanism effectively captures genuinely long-range semantic dependencies.
  4. Limited competitiveness on downstream tasks. VTDS achieves 48.9 box AP on COCO under the 1× schedule and 48.6 mIoU on ADE20K, offering no consistent improvement over stronger existing approaches despite its higher computational budget.
Questions for authors
  1. What is the fundamental technical distinction between the proposed semantic lag estimator and conventional local attention mechanisms? Can the authors provide a direct comparison under matched computational budgets?
  2. How can the claimed long-range modeling capability be justified when explicit semantic retrieval is limited to 10 historical tokens?
  3. Why does VTDS require substantially more FLOPs than competing visual SSMs, and what specific advantages justify this additional computational cost?
  4. Can the authors provide parameter- and FLOP-matched comparisons, particularly against VMamba, LocalVMamba, and Spatial-Mamba, to establish the actual benefits of the proposed components?
1 comment

Thank you for the detailed feedback. We agree that the current manuscript overstates both the efficiency and long-range aspects, and that the novelty should be stated more precisely.

Q1. How is SLE different from conventional local attention? Can you provide matched-budget comparisons? The distinction is architectural rather than the similarity operator itself. SLE does use similarity-weighted aggregation within a bounded history window, but it is not the primary token mixer. VTDS retains full-sequence recurrent propagation and uses SLE as an explicit semantic retrieval path that selects which historical tokens should provide corrective context. The fuzzy regulator then controls how much this retrieved context contributes relative to the recurrent state.

Thus, our contribution is not a new similarity function or gating primitive in isolation, but their integration as an explicit semantic-dependency pathway inside a recurrent visual SSM. We will revise the novelty claim accordingly.

We also agree that matched-budget comparisons are necessary. We will add controlled comparisons with VMamba, LocalVMamba, and Spatial-Mamba where comparable configurations can be obtained, and clearly separate matched comparisons from results reported under different budgets.

Q2. How can VTDS claim long-range modeling with (L=10)? We agree that the explicit SLE retrieval itself is bounded and should not be described as directly retrieving arbitrary long-range tokens. Our current wording is too broad.

The intended mechanism is that SLE provides bounded content-dependent retrieval, while the recurrent path propagates information through the full serialized sequence. Alternating serialization across layers further changes which spatial regions become nearby in sequence space. Therefore, long-range information is carried primarily by recurrent propagation, with SLE serving as a semantic correction rather than a standalone long-range retrieval mechanism.

We will revise the paper to distinguish long-range recurrent propagation from bounded semantic retrieval, and avoid attributing long-range capability to the (L=10) window alone.

Q3. Why does VTDS require substantially more FLOPs, and what justifies the cost? We agree that the current VTDS configuration is compute-heavy: it requires 70.0G FLOPs for 82.5% ImageNet Top-1 and is not more efficient in absolute FLOPs than VMamba-T.

The additional cost mainly comes from explicit historical candidate construction, similarity computation, and the relatively large single-scale backbone. The intended advantage is not lower absolute computation, but explicit content-dependent retrieval while preserving linear scaling with sequence length and avoiding a global (T\times T) affinity matrix.

The current downstream results suggest that this additional modeling capacity can benefit dense prediction—for example, VTDS reaches 52.0 box AP / 46.2 mask AP on COCO under the 3× schedule—but we agree that these gains do not establish a universally favorable accuracy-compute trade-off. We will narrow the efficiency claim accordingly.

Q4. Can you provide parameter- and FLOP-matched comparisons? We agree that the current comparisons do not fully establish this. Table 1 mixes substantially different compute budgets, which weakens conclusions about efficiency.

We will add parameter/FLOP information to the component ablations and report matched-budget comparisons where feasible. A complete retraining of multiple VTDS scales to exactly match all competing architectures is beyond the rebuttal period, so we will state this limitation explicitly rather than overclaiming from unmatched comparisons.

Overall, we will position VTDS more narrowly: the main contribution is explicit semantic-dependency modeling within recurrent visual state propagation, rather than a fundamentally new attention operator, unrestricted long-range retrieval, or superior absolute FLOP efficiency.

Reviewer_LCSnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VTDS, which combines windowed attention with a bidirectional SSM, in the style of ViM. To merge the windowed information into the state the architecture uses a bounded fuzzy semantic regulator. This method is applied to vision tasks (classification, detection, segmentation) showing improved performance over baselines.

Critical strengths
  • The results are acceptable, though not especially strong.
  • The ablations indicate that the proposed components work.
  • The methods were explained clearly for easy reproducibility
Critical weaknesses
  1. Missing and mismatched comparisons
    • The main ImageNet results show a large FLOPs mismatch across parameter counts.
    • Comparisons are missing against contemporary methods with similar intuition and design, such as DefMamba (CVPR 2025) and Deformba (ICML 2026).
    • Many comparisons are also missing among models in the SSM family.
    • There is no throughput comparison to contemporary models such as VSSD, Mamba2D and MambaVision.
    • No scaling results are reported, since only one model size is evaluated. This weakens comparisons to differently sized models and to prior work.
  2. Unclear efficiency claim
    • Each table seems to present a different aspect of efficiency, so it is unclear what the efficiency claim is, if there is one. Framing each comparison differently may hide weak points, which is not good practice.
  3. Insufficient justification of the architecture
    • The paper spends a lot of space describing the architecture but very little justifying why each component is included or why the design works.
  4. Ablations do not account for FLOPs
    • The ablations do not report the change in FLOPs for each component, so their true contribution is hard to judge.
Questions for authors
  1. What is the efficiency claim, if any? If there is none, why should this be used over a transformer?
  2. Why does this design work? How does it differ from similar windowed attention methods?
  3. In the ablations, what is the true advantage of each component once FLOPs are normalized?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.