- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper analyzes the representational differences between VMamba and MambaOut. The authors find that VMamba can model both local and global information, and relies more on token direction to encode class information, whereas MambaOut relies more on high-norm foreground tokens. Further analysis shows that this difference is related to the normalization–pooling order and is associated with performance in high-resolution classification and semantic segmentation. Overall, the paper offers some interpretive value, but both the presentation and the scope of validation could be further improved.
Critical strengths
-
The research question has some fundamental significance, and the analysis is relatively diverse. The paper mainly relies on experiments such as CKA and memory length.
-
The authors find that VMamba relies more on token direction to encode class information, whereas MambaOut relies more on token magnitude. This observation is insightful and is consistently validated across the Tiny, Small, and Base model scales.
Critical weaknesses
-
The Introduction is brief and lacks an intuitive conceptual illustration of the core research question. For readers who are not familiar with Vision Mamba and MambaOut, it may be difficult to understand the key differences between the two models and the motivation of this work, which increases the barrier to understanding the paper.
-
The authors identify representational differences between VMamba and MambaOut, but they have not yet demonstrated that these differences are the reason why VMamba performs better in high-resolution classification and semantic segmentation. For example, the head swap changes the locations of high-norm tokens, but the paper does not further compare downstream performance before and after the head swap.
Questions for authors
-
After changing the VMamba head from LN→GAP to GAP→LN, do the high-resolution classification and semantic segmentation results also change significantly?
-
Do the token encoding differences identified in this paper also appear in other Vision Mamba architectures?
-
Could the authors add object detection or instance segmentation experiments to further evaluate whether the conclusions generalize to other dense prediction tasks?
2 comments
Comments on the Rebuttal
Thank you to the authors for their detailed responses and additional experiments. I also value the perspectives of the other reviewers, particularly those with greater expertise in this research area.
Thank you for your feedback. We address each comment below.
As suggested, we will clarify the Introduction by providing a conceptual illustration of the key architectural differences between VMamba and MambaOut and explaining the motivation for our study.
The key architectural differences between the two models lie in how they propagate information and use it for final predictions. VMamba unfolds image feature maps into sequences along four directions and applies selective scans, allowing information to propagate between distant spatial locations within a single block. In contrast, MambaOut replaces selective scans with Gated CNNs, which primarily aggregate information from nearby regions through convolution. The models also differ in their classification heads: VMamba uses LN$\rightarrow$GAP, which normalizes each token before pooling, whereas MambaOut uses GAP$\rightarrow$LN, which normalizes the pooled features.
Despite these architectural differences, MambaOut achieves comparable or higher performance than VMamba on standard image classification, whereas VMamba shows relative advantages in segmentation and object detection. This raises a central question: why do the two models achieve similar standard image classification performance but differ in high-resolution and dense prediction tasks?
To address this question, we analyzed the internal representations from two perspectives: how far information propagates and how class information is encoded in and utilized through tokens.
Following the reviewer's suggestion, we conducted additional head-swap experiments during the rebuttal period. We applied GAP$\rightarrow$LN and LN$\rightarrow$GAP heads to VMamba-T and MambaOut-T, respectively, and retrained both models on ImageNet-1K. We then evaluated all four configurations under the same evaluation conditions on ObjectNet zero-shot classification and Pascal VOC 2012 segmentation fine-tuning.
As shown in Table 1, VMamba maintained higher performance than MambaOut on high-resolution classification and segmentation even after changing the classification-head order. On ObjectNet at 768 resolution, when both models used the same head order, VMamba outperformed MambaOut by 3.83 pp with LN$\rightarrow$GAP and 8.09 pp with GAP$\rightarrow$LN. On Pascal VOC 2012, VMamba also achieved higher mIoU by 3.02 and 3.19 pp, respectively.
These results show that VMamba consistently outperforms MambaOut on ObjectNet 768 and Pascal VOC 2012, supporting the interpretation that the backbone representation properties observed in the paper are related to downstream performance.
Table 1. Head-swap comparison
VOC 2012: UperNet full fine-tuning, mean ± SD over three seeds. ObjectNet: zero-shot evaluation, top-1 accuracy (%).
Following the reviewer's suggestion, we plan to extend our evaluation to object detection and instance segmentation. As reported in our response to W2, we have also conducted additional Pascal VOC 2012 semantic segmentation experiments with three random seeds, which provide further evidence of VMamba's advantages on dense prediction tasks. The object detection and instance segmentation experiments are expected to require approximately 3–4 days and 2–3 days, respectively. If the results are not available within the rebuttal period, we will include them in the camera-ready version.