acceptodds
Under review as a conference paper at ICLR 2027

MindViT: A Hybrid Vision Backbone with Adaptive Bidirectional Mamba

Abstract

While Mamba-based models have shown strong potential for long sequence modeling, adapting them to vision is challenging due to the requirement of local neighborhood correlations and multi-directional spatial contexts for visual understanding. In this paper, we present MindViT, a novel hybrid vision backbone that synergizes the strengths of CNNs, Mamba, and Transformers. The core of our design is the Adaptive Bidirectional Mamba (ABM), which mitigates the directional bias of unidirectional SSMs through bidirectional selective scanning with learnable alpha blending, enabling content-adaptive directional fusion without the overhead of exhaustive multi-path routing. To further accelerate inference, we propose a deployment-friendly Reparameterized ConvSE (RepConvSE) module that leverages structural reparameterization to reduce latency and memory access overhead. Extensive experiments demonstrate that MindViT achieves strong performance on image classification, object detection, and semantic segmentation while enabling efficient inference through reparameterization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Hybrid Architecture, Vision Backbone, Representation Learning, State Space Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UPfUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a hybrid CNN–Mamba–Transformer architecture for various vision tasks. Its main contribution is an alpha-blending mechanism that adapts bidirectional Mamba to image inputs. It also introduces RepConvSE, an engineering refinement of the CNN block designed to facilitate deployment.

Critical strengths
  1. The paper provides extensive experimental evaluations across a wide range of downstream scenarios.
  2. The proposed modifications are lightweight, and the method consistently outperforms all compared baselines across the evaluated downstream tasks.
Critical weaknesses
  1. The motivation of this paper is unclear. As I understand, this paper appears primarily focused on combining CNN, Mamba, and Transformer components, without clearly identifying the problem that motivates this design.

  2. The hybrid architectural novelty is limited. Existing methods, including MambaVision, which is included in the comparisons, have already explored hybrid architectures incorporating Mamba.

  3. RepConvSE is a deployment optimization based on existing techniques. There is not enough evidence which can prove its efficiency and novelty. The experiments also lack comparisons with similar optimization approaches, such as RepMBConv.

  4. In Section 4.1, this paper states that "all compared methods were trained for 300 epochs". I am wondering which methods does “all compared methods” refer to? Some reported results exactly match those in the original or other published papers, although their original experimental configurations differ from the setup stated in this paper. This raises concerns about the consistency of the reported experimental settings and the validity of the comparisons.

  5. Table 4 could not support the efficiency of the two main contributions, alpha blending and RepConvSE. Adding them can provide only marginal improvements. Even without either component, the model achieves 83.2% accuracy, ranking second only to MambaVision-S (-0.1%) in Table 1. More concerningly, the architecture of MambaVision is highly similar to the proposed model.

Questions for authors
  1. Please address the concern raised in Weakness 3
  2. Please provide additional ablation comparisons for RepConvSE to substantiate its novelty and effectiveness. Please compare it with existing optimization methods, such as RepMBConv, and clarify why existing approaches are insufficient.
  3. Please provide ablation results across different downstream tasks, as the current results (Table 4) do not clearly demonstrate the benefits of the proposed components.
Reviewer_GbASOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MindViT is a hierarchical backbone combining convolutional processing at high spatial resolution, an SSD/Mamba-based third stage, and LaPE Transformer blocks in the final stage. Its two principal contributions are Adaptive Bidirectional Mamba (ABM), which partitions SSM heads between forward and reverse scans and mixes corresponding directional outputs using input-dependent coefficients, and RepConvSE, an affine convolutional channel mixer with learned, input-independent channel scales that can be collapsed for deployment.

Critical strengths
  1. Restricting attention to the final, low-resolution stage and using reparameterizable convolutional components earlier is a sensible way to combine local processing, longer-range aggregation, and deployment efficiency. The proposed blend adds a small, understandable mechanism to a bidirectional SSD branch.

  2. Tables 2 and 3 provide evidence that the learned features transfer to detection, instance segmentation, and semantic segmentation. The Base results are particularly interesting: relative to the listed MambaVision-B results, the manuscript reports improvements of 1.5 box AP, 1.1 mask AP, and 0.5 mIoU, alongside lower parameter counts where reported.

  3. Table 4 separates replacing ConvFFN, introducing bidirectionality, and adding alpha blending. The improvement from 83.6% to 83.9% after enabling blending is relevant evidence, and Table 5 explicitly quantifies the accuracy/throughput trade-off when adding scan routes. I appreciate that four paths do not improve accuracy and eight improve it only marginally, rather than every additional component being presented as uniformly beneficial.

Critical weaknesses
  • Section 3.2 says that heads are evenly partitioned into forward and backward groups, followed by pairwise operations in Equation (4). However, Table 6 lists five ABM heads for MindViT-B. Five cannot be divided into two equal groups without an additional convention. Separately, the Small model's Stage 4 lists channel width 512 and 12 attention heads, i.e., a standard equal-width split of 512 across 12 heads is impossible unless the attention projection has a different internal width.

  • It compares unblended bidirectional outputs with learned alpha blending, but includes neither fixed symmetric averaging nor an input-independent learned blend. From Equation (4), the sum of each output pair is always Y_fwd + Y_bwd, while its difference becomes (2 alpha - 1) (Y_fwd - Y_bwd). Thus the operation preserves the common component and gates the directional difference; at the proposed initialization, both output streams are identical. This is a potentially useful inductive bias, but the 0.3-point improvement could reflect this mixing or its optimization effects rather than spatial adaptation specifically. I would like to see fixed alpha = 0.5 and learned per-layer/per-head constants, with the same surrounding block. An initialization comparison would also substantiate the claimed stable optimization.

  • In Table 1, MindViT-T uses 3.4G FLOPs and 12.6M parameters, versus 1.4G/6.8M for FastViT-T12. MindViT-S uses 9.3G FLOPs versus 5.6G for FastViT-SA36 and 7.5G for MambaVision-S. The accuracy improvements are real reported outcomes, but do not by themselves establish a superior compute trade-off in those groups. The Base comparison is closer and more persuasive. Furthermore, I wonder why the pioneering works, such as Vim, VMamba, and LocalMamba, are not presented in the experiments section.

Questions for authors
  1. What are the actual SSD head counts and forward/backward group sizes in MindViT-B, and how are the five heads in Table 6 paired? What projection width allows 12 attention heads at Stage 4 of MindViT-S?

  2. How does adaptive alpha compare with frozen alpha = 0.5 and learned input-independent per-head coefficients? Does the benefit persist across seeds, and what happens with an initialization closer to concatenation?

  3. Were the Table 1 baselines retrained under the stated recipe? Please specify each baseline's input resolution, validation crop, and the resolution used for FLOP counting, together with the source of each reported accuracy.

Reviewer_fZcBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MindViT, a hybrid vision backbone that combines CNNs, Mamba, and Transformers in a four-stage hierarchical framework. The two main technical contributions are Adaptive Bidirectional Mamba (ABM), which fuses the two directional outputs using input-dependent per-token alpha blending coefficients, instead of simple concatenation, and RepConvSE, which introduces learnable scaling parameters in a reparameterization block. Experiments on ImageNet-1K classification, MS-COCO detection/instance segmentation, and ADE20K semantic segmentation show competitive accuracy-efficiency trade-offs.

Critical strengths
  • Strong performance: The method achieves strong performance on standard benchmarks, including ImageNet-1K, COCO, and ADE20K, demonstrating its practical effectiveness.
Critical weaknesses
  • Incremental novelty: While adaptively combining the scans of two paths via alpha blending seems reasonable, it is unclear whether the same adaptive merging could be achieved using conventional concatenation followed by a feed-forward layer. Regarding RepConvSE, it appears that the method does not make channel attention reparameterizable; rather, it simply introduces learnable scaling parameters that can be reparameterized.
  • Lack of ablation studies: I do not think the paper provides sufficient ablation studies to demonstrate the effectiveness of the proposed ABM and RepConvSE modules. Although RepConvSE is compared with ConvFFN, a direct comparison between RepConv and RepConvSE should also be conducted. Similarly, for ABM, the authors should evaluate how alpha blending compares with conventional alternatives such as concatenation and addition.
Questions for authors
  • As mentioned in the weaknesses, I would like to see ablation studies that demonstrate the differences between the proposed modules and similar existing modules, rather than simply evaluating whether to use RepConvSE and ABM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.