- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper observes the mismatch in sequential distance to semantic relevance, which occurs when a 2D image is serialized into a 1D sequence. The paper tackles this mismatch with the proposed VTDS framework, which works by extracting the relevant information within a bounded window to then fusing it with the information gathered through a simplified diagonal recurrence using a fuzzy semantic regulator with enhanced spatial coverage via alternating spatial scans across layers. The experiments show that VTDS achieves competitive performance across datasets and tasks, although with a noticeably higher compute cost than existing visual SSMs.
Critical strengths
- The paper addresses a relevant problem in the field, where converting a 2D image into a 1D sequence with a fixed serialization order leads to a clear mismatch between the spatial and semantic relationships. The semantic retrieval mechanism proposed to address this problem is an interesting approach which seems to complement well with the state propagation recurrence and with the added learnable retrieval distribution seems to provide with some interpretability of the model's behavior.
- A clear ablation study has been performed to justify the design choice, as it shows a clear incremental increase in performance with the addition of all the proposed components, especially with the largest improvement coming from the semantic lag estimator.
- The model evaluated across various vision tasks suggests some potential benefits for the proposed framework on dense prediction tasks, despite the larger compute budget.
Critical weaknesses
-
While the core motivation of the paper being long-range semantic dependency modeling, the proposed framework does not seem to sufficiently support it. The recurrence is modeled to add semantic relations with the use of a semantic lag estimator, but with a very limited fixed history (L=10), it covers only a very limited portion, especially when considering larger images. While the overall information coverage could be considered to be longer with the use of alternating directions and recurrent propagation, the proposed semantic retrieval remains local. This raises a question of whether the improvements come from local-content-based aggregation or long-range semantic modeling.
-
Architectural Novelty and trade-offs are not sufficiently justified with the use of existing approaches. Works like LocalMamba (Huang et al., 2024) have already explored the serialization and scan-direction limitation, while many other works like Mamba-Adaptor (Xie et al., 2025) and TV3S (Hesham et al., 2025) address the serialization problem for long-range modeling. Furthermore, the use of a semantic lag estimator for close-range content assessment seems to align with local content-based attention that has been explored in methods like Neighbourhood Attention (Hassani et al., 2023). VTDS indeed simplifies Mamba's content-selectiveness with the use of fixed diagonal recurrence to allow for parallelization, but adds a retrieval and fuzzy attention mechanism that increases the compute. It is not clear in the current state whether the advantage coming from the added complexity is meaningful over a straightforward use of local attention + recurrence or maybe just the existing selective SSM.
-
The accuracy-efficient claim in the paper is not convincingly demonstrated in the experimental results. Table 1 clearly shows the higher requirement of VTDS to achieve similar performance to Mamba-T, and is similarly seen in the other results. While Table 4 shows some favorable throughput, its FLOP count seems to differ from the Table1, the batch size and throughput seems inconsistent. This makes the claims in the paper difficult to assess.
Questions for authors
-
With the proposed retrieval having a limited and fixed historical window, how can the authors demonstrate that the reported improvement comes from the semantic relations rather than local feature aggregation? And how can the mechanism remain effective as the image resolution, and hence the sequence lengths, increases beyond the limited retrieval window (L=10)?
-
What is the fundamental advantage of the proposed methods, which use semantic retrieval in local windows, over existing selective SSMs or a simpler local-attention+recurrence architecture? A controlled comparison would really help clarify the novelty of the proposed design.
-
How do the authors justify the claim made in the paper on the accuracy-efficiency trade-offs when the proposed methods have substantially higher compute and parameters than the current SSMs? Could the authors clarify the inconsistencies in the results in Tables 1 and 4?
Thank you for the detailed feedback. We agree that the current manuscript overstates both the efficiency and long-range aspects, and that the novelty should be stated more precisely.
Q1. How is SLE different from conventional local attention? Can you provide matched-budget comparisons? The distinction is architectural rather than the similarity operator itself. SLE does use similarity-weighted aggregation within a bounded history window, but it is not the primary token mixer. VTDS retains full-sequence recurrent propagation and uses SLE as an explicit semantic retrieval path that selects which historical tokens should provide corrective context. The fuzzy regulator then controls how much this retrieved context contributes relative to the recurrent state.
Thus, our contribution is not a new similarity function or gating primitive in isolation, but their integration as an explicit semantic-dependency pathway inside a recurrent visual SSM. We will revise the novelty claim accordingly.
We also agree that matched-budget comparisons are necessary. We will add controlled comparisons with VMamba, LocalVMamba, and Spatial-Mamba where comparable configurations can be obtained, and clearly separate matched comparisons from results reported under different budgets.
Q2. How can VTDS claim long-range modeling with (L=10)? We agree that the explicit SLE retrieval itself is bounded and should not be described as directly retrieving arbitrary long-range tokens. Our current wording is too broad.
The intended mechanism is that SLE provides bounded content-dependent retrieval, while the recurrent path propagates information through the full serialized sequence. Alternating serialization across layers further changes which spatial regions become nearby in sequence space. Therefore, long-range information is carried primarily by recurrent propagation, with SLE serving as a semantic correction rather than a standalone long-range retrieval mechanism.
We will revise the paper to distinguish long-range recurrent propagation from bounded semantic retrieval, and avoid attributing long-range capability to the (L=10) window alone.
Q3. Why does VTDS require substantially more FLOPs, and what justifies the cost? We agree that the current VTDS configuration is compute-heavy: it requires 70.0G FLOPs for 82.5% ImageNet Top-1 and is not more efficient in absolute FLOPs than VMamba-T.
The additional cost mainly comes from explicit historical candidate construction, similarity computation, and the relatively large single-scale backbone. The intended advantage is not lower absolute computation, but explicit content-dependent retrieval while preserving linear scaling with sequence length and avoiding a global (T\times T) affinity matrix.
The current downstream results suggest that this additional modeling capacity can benefit dense prediction—for example, VTDS reaches 52.0 box AP / 46.2 mask AP on COCO under the 3× schedule—but we agree that these gains do not establish a universally favorable accuracy-compute trade-off. We will narrow the efficiency claim accordingly.
Q4. Can you provide parameter- and FLOP-matched comparisons? We agree that the current comparisons do not fully establish this. Table 1 mixes substantially different compute budgets, which weakens conclusions about efficiency.
We will add parameter/FLOP information to the component ablations and report matched-budget comparisons where feasible. A complete retraining of multiple VTDS scales to exactly match all competing architectures is beyond the rebuttal period, so we will state this limitation explicitly rather than overclaiming from unmatched comparisons.
Overall, we will position VTDS more narrowly: the main contribution is explicit semantic-dependency modeling within recurrent visual state propagation, rather than a fundamentally new attention operator, unrestricted long-range retrieval, or superior absolute FLOP efficiency.