acceptodds
Under review as a conference paper at ICLR 2027

Norm or Direction? Decoding Vision Mambas for High-Resolution Vision

Abstract

Vision Mamba models replace quadratic self-attention with linear-complexity selective state space models (SSMs). MambaOut, however, shows that a Gated CNN block can match or exceed VMamba on image classification, questioning whether SSMs are necessary for vision. We ask whether the two models nonetheless encode visual information differently. Using centered kernel alignment (CKA), we find that VMamba's final stage diverges from both MambaOut and its own preceding blocks, and we trace this divergence to two components. The token mixer sets how far information spreads. VMamba's SS2D combines local channels with channels whose memory spans the entire image, some of which remain global at without retraining. The classifier head sets which part of a token the prediction reads. MambaOut pools before normalizing and concentrates class evidence in high-norm foreground tokens, whereas VMamba normalizes before pooling, places high-norm tokens in the background, and preserves class evidence in token directions. Swapping only the head order moves VMamba's high-norm tokens to the foreground. Consistent with direction-based evidence aggregating more stably as tokens grow, VMamba spreads logit support more broadly, degrades less under resolution transfer across three model scales, and outperforms MambaOut on segmentation after full fine-tuning. We suggest token mixing range, token magnitude and direction, and the normalization–pooling order as design axes for high-resolution and dense prediction backbones.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Compter Vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1o14OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper analyzes the representational differences between VMamba and MambaOut. The authors find that VMamba can model both local and global information, and relies more on token direction to encode class information, whereas MambaOut relies more on high-norm foreground tokens. Further analysis shows that this difference is related to the normalization–pooling order and is associated with performance in high-resolution classification and semantic segmentation. Overall, the paper offers some interpretive value, but both the presentation and the scope of validation could be further improved.

Critical strengths
  1. The research question has some fundamental significance, and the analysis is relatively diverse. The paper mainly relies on experiments such as CKA and memory length.

  2. The authors find that VMamba relies more on token direction to encode class information, whereas MambaOut relies more on token magnitude. This observation is insightful and is consistently validated across the Tiny, Small, and Base model scales.

Critical weaknesses
  1. The Introduction is brief and lacks an intuitive conceptual illustration of the core research question. For readers who are not familiar with Vision Mamba and MambaOut, it may be difficult to understand the key differences between the two models and the motivation of this work, which increases the barrier to understanding the paper.

  2. The authors identify representational differences between VMamba and MambaOut, but they have not yet demonstrated that these differences are the reason why VMamba performs better in high-resolution classification and semantic segmentation. For example, the head swap changes the locations of high-norm tokens, but the paper does not further compare downstream performance before and after the head swap.

Questions for authors
  1. After changing the VMamba head from LN→GAP to GAP→LN, do the high-resolution classification and semantic segmentation results also change significantly?

  2. Do the token encoding differences identified in this paper also appear in other Vision Mamba architectures?

  3. Could the authors add object detection or instance segmentation experiments to further evaluate whether the conclusions generalize to other dense prediction tasks?

2 comments

Thank you for your feedback. We address each comment below.

W1. The Introduction is brief and lacks an intuitive conceptual illustration of the core research question. For readers who are not familiar with Vision Mamba and MambaOut, it may be difficult to understand the key differences between the two models and the motivation of this work, which increases the barrier to understanding the paper.

As suggested, we will clarify the Introduction by providing a conceptual illustration of the key architectural differences between VMamba and MambaOut and explaining the motivation for our study.

The key architectural differences between the two models lie in how they propagate information and use it for final predictions. VMamba unfolds image feature maps into sequences along four directions and applies selective scans, allowing information to propagate between distant spatial locations within a single block. In contrast, MambaOut replaces selective scans with Gated CNNs, which primarily aggregate information from nearby regions through convolution. The models also differ in their classification heads: VMamba uses LN$\rightarrow$GAP, which normalizes each token before pooling, whereas MambaOut uses GAP$\rightarrow$LN, which normalizes the pooled features.

Despite these architectural differences, MambaOut achieves comparable or higher performance than VMamba on standard image classification, whereas VMamba shows relative advantages in segmentation and object detection. This raises a central question: why do the two models achieve similar standard image classification performance but differ in high-resolution and dense prediction tasks?

To address this question, we analyzed the internal representations from two perspectives: how far information propagates and how class information is encoded in and utilized through tokens.


W2. The authors identify representational differences between VMamba and MambaOut, but they have not yet demonstrated that these differences are the reason why VMamba performs better in high-resolution classification and semantic segmentation. For example, the head-swap changes the locations of high-norm tokens, but the paper does not further compare downstream performance before and after the head-swap.

Following the reviewer's suggestion, we conducted additional head-swap experiments during the rebuttal period. We applied GAP$\rightarrow$LN and LN$\rightarrow$GAP heads to VMamba-T and MambaOut-T, respectively, and retrained both models on ImageNet-1K. We then evaluated all four configurations under the same evaluation conditions on ObjectNet zero-shot classification and Pascal VOC 2012 segmentation fine-tuning.

As shown in Table 1, VMamba maintained higher performance than MambaOut on high-resolution classification and segmentation even after changing the classification-head order. On ObjectNet at 768 resolution, when both models used the same head order, VMamba outperformed MambaOut by 3.83 pp with LN$\rightarrow$GAP and 8.09 pp with GAP$\rightarrow$LN. On Pascal VOC 2012, VMamba also achieved higher mIoU by 3.02 and 3.19 pp, respectively.

These results show that VMamba consistently outperforms MambaOut on ObjectNet 768 and Pascal VOC 2012, supporting the interpretation that the backbone representation properties observed in the paper are related to downstream performance.

Table 1. Head-swap comparison

BackboneHeadVOC 2012 mIoUObjectNet 224ObjectNet 384ObjectNet 768
VMamba-TLN$\rightarrow$GAP (before)82.03±0.2082.03 \pm 0.2045.3345.2631.44
MambaOut-TGAP$\rightarrow$LN (before)79.27±0.4279.27 \pm 0.4245.8545.8529.56
VMamba-TGAP$\rightarrow$LN (after)82.46±0.2382.46 \pm 0.2346.3247.5937.65
MambaOut-TLN$\rightarrow$GAP (after)79.01±0.0679.01 \pm 0.0645.2745.7127.61

VOC 2012: UperNet full fine-tuning, mean ±\pm SD over three seeds. ObjectNet: zero-shot evaluation, top-1 accuracy (%).


Q1. Could the authors add object detection or instance segmentation experiments to further evaluate whether the conclusions generalize to other dense prediction tasks?

Following the reviewer's suggestion, we plan to extend our evaluation to object detection and instance segmentation. As reported in our response to W2, we have also conducted additional Pascal VOC 2012 semantic segmentation experiments with three random seeds, which provide further evidence of VMamba's advantages on dense prediction tasks. The object detection and instance segmentation experiments are expected to require approximately 3–4 days and 2–3 days, respectively. If the results are not available within the rebuttal period, we will include them in the camera-ready version.

Reviewer_1o14OpenReview ↗

Comments on the Rebuttal

Thank you to the authors for their detailed responses and additional experiments. I also value the perspectives of the other reviewers, particularly those with greater expertise in this research area.

Reviewer_YkwyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares VMamba and MambaOut to study how visual backbones encode class-discriminative information. The authors analyze representation similarity, effective receptive fields, SSM memory lengths, token norms, and token directions. They argue that SS2D provides a mixture of local and global channels, while the normalization–pooling order in the classification head determines whether class evidence is primarily encoded in token magnitude or normalized token direction. They further relate these observations to resolution transfer and semantic segmentation performance.

Critical strengths
  1. Comprehensive empirical analysis. The paper combines multiple complementary tools, including CKA, ERF, memory-length analysis, token probing, attribution, and segmentation experiments.

  2. The intervention on the normalization–pooling order is helpful: it suggests that whether high-norm tokens align with foreground regions is largely shaped by the classification head rather than simply by the token mixer.

Critical weaknesses
  1. The main questions (the effective receptive fields and high-norm/outlier tokens) have been extensively studied in prior work. The contribution is mainly to apply and combine these analyses for VMamba and MambaOut, rather than introducing a substantially new phenomenon or method.

  2. The strongest finding appears not to be Mamba-specific. The paper is framed as decoding Vision Mambas, but its most convincing result is that the norm v.s. direction behavior is primarily determined by the LN→GAP versus GAP→LN head design. Therefore, the current Mamba-centric presentation appears somewhat confusing.

  3. The analyses do not form a clear mechanistic chain. The paper presents several different observations but the causal relationships among them remain unclear. In particular, it is not established whether global channels, head-induced encoding geometry, or their interaction explains the downstream behavior.

  4. Most central representation analyses and interventions focus on a single VMamba/MambaOut comparison, often at the Tiny scale. Although some downstream results include Small and Base models, it remains unclear whether the conclusions generalize across scales, other Vision Mamba variants, or other CNN-style token mixers.

  5. It remains unclear how the analysis should guide future backbone design and how can we learn something new from the results.

Questions for authors

Please refer to the weaknesses.

5 comments

Thank you for your feedback. We address each comment below.

W1. The main questions (the effective receptive fields and high-norm/outlier tokens) have been extensively studied in prior work. The contribution is mainly to apply and combine these analyses for VMamba and MambaOut, rather than introducing a substantially new phenomenon or method.

While ERF and high-norm token analyses revealed initial patterns, they do not capture the channel-wise information propagation of SS2D or the architectural factors shaping high-norm token distributions. To investigate these aspects, we introduced a channel-wise memory-length measure and analyzed the normalization–pooling order of the classification head. Through these analyses, we identified the coexistence of local and global channels within SS2D and the relationship between head order and the spatial distribution of high-norm tokens.

  • Beyond Effective Receptive Field: ERF analysis reveals how far information propagates spatially but does not explain which channels within SS2D retain long-range information. To investigate this, we introduced channel-wise memory length and identified the coexistence of local channels that process nearby information and global channels that retain long-range information within SS2D.

  • Beyond High-norm Token Observations: The properties and roles of high-norm tokens reported in prior studies [1, 2] do not appear in the same form in VMamba. Prior studies on ViT and Vim, which use a class token for classification, reported that some background tokens acquire unusually large norms and serve as repositories for global information. These studies proposed register tokens to mitigate this behavior.

    In contrast, VMamba uses pooling instead of a class token, and its token-norm distribution does not exhibit a separate outlier mode (Fig. 5). Moreover, its highest-norm tokens do not contain the most class information (Fig. 6). Thus, VMamba's high-norm background tokens exhibit properties distinct from the outliers or global information repositories described in prior work, making it difficult to apply the same interpretation directly. Building on these distinctions, our study identifies the relationship between VMamba's high-norm token distribution and the normalization–pooling order of the classification head.

These findings suggest that understanding the internal representations of Vision Mamba requires distinguishing the information propagation properties of the token mixer from the representational effects of the classification head.


W2. The strongest finding appears not to be Mamba-specific. The paper is framed as decoding Vision Mambas, but its most convincing result is that the norm v.s. direction behavior is primarily determined by the LN$\rightarrow$GAP versus GAP$\rightarrow$LN head design. Therefore, the current Mamba-centric presentation appears somewhat confusing.

Our analyses of the factors shaping token representations can be divided into two parts: the Mamba-specific memory structure of SS2D and the classification head. The former, through the channel-wise memory-length analysis in Sec. 4.2 and Appendix C, identifies the coexistence of local and global channels within SS2D and its ability to retain long-range information at high resolutions. The latter, through analyses of token magnitude and direction (Sec. 4.3), head order (Sec. 4.4), and token attribution (Sec. 5.1), identifies how class information is distributed across tokens and how head order affects high-norm token distributions.

We will revise the camera-ready version to more clearly distinguish the roles and contributions of these two analyses.

W3. The analyses do not form a clear mechanistic chain. The paper presents several different observations but the causal relationships among them remain unclear. In particular, it is not established whether global channels, head-induced encoding geometry, or their interaction explains the downstream behavior.

Our analyses suggest that SS2D contributes to long-range information propagation within the backbone, while the normalization–pooling order of the classification head may affect how token information is aggregated and how representations are learned. To clarify the roles and relationship of these two components, we conducted additional memory-length restriction and head-swap experiments during the rebuttal period. The results show that SS2D's long-range memory contributes to downstream performance and that this contribution persists across different normalization–pooling orders of the classification head.

  • Contribution of SS2D's long-range memory to downstream performance: In Sec. 4.2, we identified the coexistence of local and global channels within SS2D and observed that some global channels retain long-range memory even at high resolutions. To evaluate the performance contribution of this property, we conducted additional experiments in which we imposed an upper bound on memory length (τ\tau) at inference time while keeping the trained VMamba weights fixed. We considered five settings relative to the feature-map width WW: 4W4W, 2W2W, WW, W/2W/2, and W/4W/4 (Table 1).

    As shown in Table 1, performance on both ObjectNet and Pascal VOC 2012 declined monotonically as the memory-length upper bound decreased. In particular, at τmax⁡=W\tau_{\max} = W, ObjectNet accuracy decreased by 1.68 pp at 224 resolution and by 4.78 pp at 768 resolution, indicating a larger performance drop at higher resolution. Pascal VOC 2012 performance also decreased from 82.26 to 81.13 mIoU.

    Moreover, VMamba's accuracy advantage over MambaOut on ObjectNet 768 narrowed to +0.27 pp at τmax⁡=4W\tau_{\max} = 4W and reversed at τmax⁡≤2W\tau_{\max} \le 2W. These results support the contribution of SS2D's long-range information retention to VMamba's high-resolution classification and dense prediction performance.

Table 1. Downstream performance under memory-length restrictions

Modelτmax⁡\tau_{\max}ObjectNet 224ObjectNet 768VOC 2012 mIoU
VMamba-T–45.3331.4482.26
VMamba-T4W4W44.7829.8382.06
VMamba-T2W2W44.2328.5281.61
VMamba-TWW43.6526.6681.13
VMamba-TW/2W/242.3724.0980.29
VMamba-TW/4W/439.0619.6078.91
MambaOut-T–45.8529.5679.27

ObjectNet: zero-shot top-1 accuracy (%). VOC 2012: mIoU (%).

  • Contribution of long-range memory across different classification-head readouts: LN$\rightarrow$GAP normalizes each token before averaging, reducing the influence of the original token norms, whereas GAP$\rightarrow$LN averages tokens before normalization, preserving differences in their contributions due to token norms. To compare the effects of long-range memory under these two readout mechanisms, we applied the same memory-length restrictions to VMamba models retrained with swapped head orders during the rebuttal period (Table 2).

    As shown in Table 2, although the two heads achieved different baseline accuracies, their performance drops under memory-length restrictions were similar. At τmax⁡=4W\tau_{\max} = 4W, accuracy decreased by 1.61 and 1.66 pp, respectively, while at τmax⁡=W\tau_{\max} = W, the decreases were 4.78 and 4.55 pp. These results show that the contribution of SS2D's long-range memory to high-resolution performance persists across different classification-head readout mechanisms.

Table 2. Effect of memory-length restrictions under different head orders (ObjectNet 768, top-1 accuracy, %)

HeadUnrestrictedτmax⁡=4W\tau_{\max} = 4Wτmax⁡=W\tau_{\max} = W
LN$\rightarrow$GAP31.4429.83 (−1.61-1.61)26.66 (−4.78-4.78)
GAP$\rightarrow$LN37.6535.99 (−1.66-1.66)33.10 (−4.55-4.55)

These results suggest that the two components can be distinguished by their respective roles in information propagation and information aggregation for final prediction.

W4. Most central representation analyses and interventions focus on a single VMamba/MambaOut comparison, often at the Tiny scale. Although some downstream results include Small and Base models, it remains unclear whether the conclusions generalize across scales, other Vision Mamba variants, or other CNN-style token mixers.

In response to the reviewer's concern, we extended our high-norm token analysis to different architectures spanning Vision Mamba, Transformers, and CNNs, as well as Tiny, Small, and Base model scales. The results show that the association between classification-head order and the spatial distribution of high-norm tokens observed in VMamba also appears across these architectures and model scales.

  • Generalization of high-norm token spatial distributions: In the VMamba head-swap experiment reported in the paper (Fig. 7(d)), we observed a relationship between head order and the spatial distribution of high-norm tokens: they were concentrated in background regions with LN$\rightarrow$GAP and in foreground regions with GAP$\rightarrow$LN. To examine whether this trend also appears in other architectures, we additionally analyzed Tiny, Small, and Base variants of ConvNeXt [3], Swin [4], MSVMamba [5], and EfficientVMamba [6].

    For this additional analysis, we used foreground masks from ImageNet-Segmentation [7] to measure the fraction of the top 10% highest-norm tokens in the last block that fall within foreground regions. The random-selection baseline was 42.2%, with higher and lower values indicating a relative concentration in foreground and background regions, respectively.

    As shown in Table 3, high-norm tokens in ConvNeXt, which uses GAP$\rightarrow$LN, were concentrated in foreground regions across all model scales. In contrast, those in Swin, MSVMamba, and EfficientVMamba, which use LN$\rightarrow$GAP, were concentrated in background regions.

Table 3. Foreground ratio of the top 10% highest-norm tokens (%, last block)

BackboneHeadTinySmallBase
ConvNeXtGAP$\rightarrow$LN79.077.478.4
Random (baseline)–42.242.242.2
SwinLN$\rightarrow$GAP30.322.615.3
MSVMambaLN$\rightarrow$GAP24.416.422.8
EfficientVMambaLN$\rightarrow$GAP9.511.36.9

W5. It remains unclear how the analysis should guide future backbone design and how can we learn something new from the results.

By analyzing the internal information propagation mechanisms of Vision Mamba and the architectural factors affecting representation formation, our study provides two considerations for future backbone design and evaluation, as discussed in Sec. 6.

First, channel-wise long-range information retention should be considered alongside receptive-field size in SS2D-based backbones. While ERF analysis reveals how far information propagates spatially, it does not explain how long individual channels within SS2D retain information. Our channel-wise memory-length analysis identified the coexistence of local and global channels, and additional memory-length restriction experiments showed that reducing long-range memory degrades high-resolution classification performance. These findings suggest that future evaluations of SSM-based backbones should examine channel-wise memory structures and assess how restricting long-range memory affects high-resolution performance.

Second, the effects of the token mixer and classification head should be distinguished when comparing backbone representations. We observed differences in the spatial distribution of high-norm tokens associated with the normalization–pooling order of the classification head. Additional experiments also revealed similar spatial distribution patterns associated with head order across different Vision Mamba, Transformer, and CNN architectures and model scales. Therefore, without controlling for the classification head, differences arising from head design may be incorrectly attributed to the token mixer when comparing representations across backbones.

While these findings provide considerations for future backbone design and evaluation, developing and experimentally validating new backbone architectures is beyond the scope of this study.

[1] Darcet et al., "Vision Transformers Need Registers," ICLR 2024.
[2] Wang et al., "Mamba-Reg: Vision Mamba Also Needs Registers," CVPR 2025.
[3] Liu et al., "A ConvNet for the 2020s," CVPR 2022.
[4] Liu et al., "Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows," ICCV 2021.
[5] Shi et al., "Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model," NeurIPS 2024.
[6] Pei et al., "EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba," AAAI 2025.
[7] Guillaumin et al., "ImageNet Auto-Annotation with Segmentation Propagation," IJCV 2014.

Reviewer_3z1kOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why VMamba continues to outperform MambaOut on dense prediction and high-resolution tasks, despite their comparable ImageNet classification performance. Through analyses of CKA similarity, effective receptive fields, and SS2D memory length, the authors identify several characteristics of VMamba: its final stage exhibits distinct behavior, its representations depend on the type of supervision, and SS2D maintains global channels even at 768×768 resolution. The paper further finds that MambaOut primarily encodes class-related information through token norms, with high-norm tokens concentrated on foreground objects, whereas VMamba relies more on token directions. The authors attribute this difference to the ordering of LayerNorm and global average pooling in the classification head. Swapping VMamba's head to GAP→LN and retraining shifts its high-norm tokens toward the foreground. Finally, the paper connects these observations to VMamba's smaller accuracy degradation on ObjectNet at higher resolutions and its stronger performance on segmentation after full fine-tuning.

Critical strengths
  1. The paper investigates a well-motivated question. While MambaOut has challenged the necessity of SSMs for visual recognition, VMamba still shows advantages on dense prediction tasks. Understanding the mechanisms behind this difference is an interesting research direction.

  2. The finding about norm-based versus direction-based information encoding is interesting and supported by multiple complementary experiments, including comparisons between token norm maps and Grad-CAM, top-k high-norm token probing, and unit-norm single-token probing.

  3. The classification head experiment provides a useful insight into the influence of architectural details. Simply changing the order of LayerNorm and global average pooling is sufficient to alter the spatial distribution of high-norm tokens, suggesting a straightforward design factor worth investigating.

  4. The SS2D memory-length analysis provides an interpretable perspective on information propagation in VMamba. Distinguishing local and global channels and examining their behavior at higher resolutions helps characterize the spatial properties of the learned representations.

  5. The analysis covers three model scales (Tiny, Small, and Base) and includes additional control experiments, such as replacing tokens with random vectors of the same norm, which strengthens the evidence for the reported observations.

Critical weaknesses
  1. The central causal explanation is not sufficiently validated. The paper suggests that direction-based encoding and global SS2D channels contribute to VMamba's advantages on high-resolution and dense prediction tasks. However, it does not evaluate the head-swapped VMamba or MambaOut with an LN→GAP head on high-resolution ObjectNet or segmentation benchmarks. The authors also acknowledge that the connection is correlational and that the downstream effects of changing the classification head remain unexplored. Moreover, the ObjectNet experiments cannot isolate the contributions of the token mixer and the classification head. Consequently, the proposed explanation for the performance differences remains largely speculative.

  2. Several reported performance differences are relatively small, without sufficient statistical evidence to establish their significance. VMamba and MambaOut differ by only 0.1 on ImageNet at 768 resolution, 0.8 mIoU on ADE20K with full fine-tuning, 0.5 mIoU with a frozen backbone, and 0.8 in the Tiny ObjectNet accuracy drop. Without multiple random seeds, confidence intervals, or significance tests, it is difficult to determine whether these differences are meaningful. Additionally, the segmentation experiments are limited to Tiny models.

  3. The comparison between VMamba and MambaOut contains several confounding factors. Beyond the token mixer and classification head, the two architectures differ in block structure (VSS with an FFN versus Gated CNN), computational cost, depth, and training recipes. These factors are not adequately controlled, making it difficult to attribute the observed differences specifically to SS2D or normalization order. Furthermore, the architecture of the "SS2D removed and retrained" variant is not clearly specified, limiting the interpretability of this comparison.

  4. Much of the supporting evidence is qualitative, while the memory-length analysis relies on simplifying assumptions. Several conclusions are mainly drawn from ERF visualizations, token norm maps, and Grad-CAM, without quantitative measurements such as the overlap between high-norm tokens and foreground segmentation masks. The memory-length estimate τ also uses an average step size, overlooking the input-dependent behavior of Δ, while the criterion τ > L for identifying global channels is heuristic. The paper does not directly establish a quantitative relationship between global-channel proportions and downstream performance.

  5. The generality and methodological contribution of the paper are relatively limited. The experiments focus exclusively on VMamba and MambaOut, leaving it unclear whether the suggested advantage of direction-based encoding at higher resolutions extends to ViTs, ConvNeXt, or other Mamba architectures. Some observations, including high-norm background tokens and the influence of normalization order, are also related to previously studied phenomena such as register tokens. Finally, the proposed design directions, including angular objectives, are not experimentally validated.

Questions for authors
  1. How does VMamba with the swapped GAP→LN head perform on ObjectNet at 384/768 resolution and on ADE20K or VOC segmentation? Conversely, what happens when MambaOut uses an LN→GAP head?

  2. Could you provide results over multiple random seeds or confidence intervals for the ImageNet, ObjectNet, and segmentation experiments, particularly given the relatively small performance differences?

  3. Could you clarify the architecture of the "SS2D removed and retrained" variant? How are other architectural differences and training configurations between VMamba and MambaOut controlled in the comparisons?

  4. Is there a measurable relationship between the proportion of global channels and performance degradation at higher resolutions, for example across the Tiny, Small, and Base models?

8 comments

Thank you for your feedback. We address each comment below.

W1 & Q1. The central causal explanation is not sufficiently validated. The paper suggests that direction-based encoding and global SS2D channels contribute to VMamba's advantages on high-resolution and dense prediction tasks. However, it does not evaluate the head-swapped VMamba or MambaOut with an LN$\rightarrow$GAP head on high-resolution ObjectNet or segmentation benchmarks. The authors also acknowledge that the connection is correlational and that the downstream effects of changing the classification head remain unexplored. Moreover, the ObjectNet experiments cannot isolate the contributions of the token mixer and the classification head. Consequently, the proposed explanation for the performance differences remains largely speculative.

Following the reviewer's suggestion, we conducted additional head-swap and SS2D memory-length restriction experiments during the rebuttal period. The results show that VMamba consistently outperforms MambaOut on high-resolution classification and segmentation even after swapping the classification head. Furthermore, the memory-length restriction experiments show that SS2D's long-range memory contributes to high-resolution performance under both head configurations.

  • Head-swap experiments: We applied GAP$\rightarrow$LN and LN$\rightarrow$GAP heads to VMamba-T and MambaOut-T, respectively, and retrained both models on ImageNet-1K for 300 epochs. We then evaluated all four configurations on ObjectNet zero-shot classification and Pascal VOC 2012 segmentation fine-tuning.

    As shown in Table 1, when both models use the same head order, VMamba outperformed MambaOut on ObjectNet 768 by 3.83 pp with LN$\rightarrow$GAP and 8.09 pp with GAP$\rightarrow$LN. On Pascal VOC 2012, VMamba also achieved higher mIoU by 3.02 and 3.19 pp, respectively. These results show that VMamba's advantage in high-resolution classification and segmentation persists despite changes in head order.

Table 1. Head-swap comparison

BackboneHeadVOC 2012 mIoUObjectNet 384ObjectNet 768
VMamba-TLN$\rightarrow$GAP82.03±0.2082.03 \pm 0.2045.26 [44.54, 45.99]31.44 [30.78, 32.10]
MambaOut-TLN$\rightarrow$GAP79.01±0.0679.01 \pm 0.0645.71 [44.99, 46.43]27.61 [26.97, 28.25]
VMamba-TGAP$\rightarrow$LN82.46±0.2382.46 \pm 0.2347.59 [46.87, 48.30]37.65 [36.95, 38.34]
MambaOut-TGAP$\rightarrow$LN79.27±0.4279.27 \pm 0.4245.85 [45.12, 46.56]29.56 [28.90, 30.21]

VOC 2012: full fine-tuning, mean ±\pm SD over three seeds. ObjectNet: zero-shot evaluation, top-1 accuracy (%) with 95% bootstrap confidence intervals over test images.

  • Contribution of SS2D's long-range memory: To evaluate the contribution of SS2D's long-range memory independently of the classification head, we conducted an additional memory-length restriction experiment during the rebuttal period. In Sec. 4.2, we defined the memory length (τ\tau) of each scan based on the state decay in SS2D. A larger τ\tau indicates that information from preceding tokens is retained over longer distances. Thus, restricting the memory length allows us to evaluate how reduced long-range information retention affects performance. Specifically, we applied the same memory-length restrictions to VMamba models separately retrained with LN$\rightarrow$GAP and GAP$\rightarrow$LN heads. For each model, we kept the trained weights and classification head fixed and imposed an upper bound (τmax⁡\tau_{\max}) on the memory length at inference time. We then compared the changes in ObjectNet 768 accuracy before and after the restriction. τmax⁡\tau_{\max} was defined relative to the feature-map width WW.

    As shown in Table 2, although the two heads achieved different baseline accuracies, their performance drops under memory-length restrictions were nearly identical. At τmax⁡=4W\tau_{\max} = 4W, accuracy decreased by 1.61 and 1.66 pp, respectively; at τmax⁡=W\tau_{\max} = W, the decreases were 4.78 and 4.55 pp. These results support the contribution of SS2D's long-range memory to performance under both head configurations, distinct from the classification head's readout mechanism.

Table 2. Effect of memory-length restrictions under different head configurations (ObjectNet 768, top-1 accuracy, %)

HeadUnrestrictedτmax⁡=4W\tau_{\max} = 4Wτmax⁡=W\tau_{\max} = W
LN$\rightarrow$GAP31.4429.83 (−1.61-1.61)26.66 (−4.78-4.78)
GAP$\rightarrow$LN37.6535.99 (−1.66-1.66)33.10 (−4.55-4.55)

W2 & Q2. Several reported performance differences are relatively small, without sufficient statistical evidence to establish their significance. VMamba and MambaOut differ by only 0.1 on ImageNet-1K at 768 resolution, 0.8 mIoU on ADE20K with full fine-tuning, 0.5 mIoU with a frozen backbone, and 0.8 in the Tiny ObjectNet accuracy drop. Without multiple random seeds, confidence intervals, or significance tests, it is difficult to determine whether these differences are meaningful. Additionally, the segmentation experiments are limited to Tiny models.

As suggested by the reviewer, we computed 95% confidence intervals for ObjectNet evaluation and repeated Pascal VOC 2012 segmentation experiments with three random seeds to assess the reliability of the reported performance differences. The resulting confidence intervals support VMamba's performance advantage on ObjectNet, while the performance differences between backbones on Pascal VOC 2012 exceed the variation across random seeds. For a more rigorous comparison, we additionally retrained VMamba-T and MambaOut-T under the same conditions. The results are summarized in Table 3.

Table 3. Downstream performance and statistical variability of VMamba-T and MambaOut-T

BackboneVOC 2012 mIoUObjectNet 224ObjectNet 768
VMamba-T82.03±0.2082.03 \pm 0.2045.33 [44.61, 46.02]31.44 [30.78, 32.10]
MambaOut-T79.27±0.4279.27 \pm 0.4245.85 [45.11, 46.55]29.56 [28.90, 30.21]

VOC 2012: full fine-tuning, mean ±\pm SD over three seeds. ObjectNet: zero-shot evaluation, top-1 accuracy (%) with 95% bootstrap confidence intervals over test images.

  • Small performance difference on ObjectNet 768 (95% bootstrap CI): The accuracy difference between the retrained models on ObjectNet at 768 resolution was 1.88 pp (paired bootstrap 95% CI: [1.30, 2.45]), and their respective 95% confidence intervals, [30.78, 32.10] and [28.90, 30.21], did not overlap. Moreover, as the input resolution increased from 224 to 768, VMamba's accuracy dropped by 13.89 pp, which is 2.40 pp (paired bootstrap 95% CI: [1.62, 3.14]) less than MambaOut's drop of 16.29 pp. This supports VMamba's high-resolution accuracy advantage even when accounting for sampling variability across evaluation images.
  • ADE20K performance differences and statistical significance (3 seeds): On Pascal VOC 2012, the performance difference between the two models was 2.76 mIoU, exceeding the standard deviations across seeds (0.20 and 0.42, respectively). We also plan to conduct a three-seed evaluation on ADE20K, which is expected to require approximately 7–8 days of training. If the results cannot be obtained during the rebuttal period, we will include them in the camera-ready version.
  • The 0.1 pp difference on ImageNet-1K 768: We acknowledge the reviewer's concern regarding the small performance difference on ImageNet-1K at 768 resolution. As our primary focus is on downstream tasks, we did not conduct a separate statistical significance test for this difference. Instead, we focus our main claims on the performance advantages observed on ObjectNet at 768 resolution and Pascal VOC 2012, where additional experiments provide stronger statistical evidence.

We also plan to extend the Pascal VOC 2012 segmentation experiments to the Small and Base models, which will require approximately 3–4 additional days of training. If these results are not available during the rebuttal period, we will include them in the camera-ready version.

W3 & Q3. The comparison between VMamba and MambaOut contains several confounding factors. Beyond the token mixer and classification head, the two architectures differ in block structure (VSS with an FFN versus Gated CNN), computational cost, depth, and training recipes. These factors are not adequately controlled, making it difficult to attribute the observed differences specifically to SS2D or normalization order. Furthermore, the architecture of the "SS2D removed and retrained" variant is not clearly specified, limiting the interpretability of this comparison.

As the reviewer noted, VMamba and MambaOut differ not only in their token mixers and classification heads but also in block structure, depth, and training configurations. To control for these differences, we conducted experiments within the VMamba architecture, modifying only the component under investigation. The SS2D memory-length analysis (Sec. 4.2) used the same trained weights, while the effect of normalization–pooling order (Fig. 7(d)) was examined by modifying only the classification head within the same VMamba backbone. These experiments provide evidence beyond a direct comparison between the two architectures.

Additionally, "SS2D removed and retrained" refers to a VMamba-T variant in which only the selective scan module is removed from each VSS block. The input projection, 3×33 \times 3 depthwise convolution, and FFN are retained. We will clarify the architecture of this variant in the revised paper.

Comments on the paper

1 comment

In response to the reviewers' feedback, we have incorporated the additional experiments conducted during the rebuttal process into the revised manuscript and updated the corresponding explanations. The main revisions are summarized below.

  1. Representation analysis (Secs. 4.3–4.4): We added quantitative analyses of the foreground overlap of high-norm tokens and unit-vector probing results for the head-swapped models. We also revised our interpretation to distinguish the roles of the backbone and classification head.
  2. High-resolution evaluation (Sec. 5.1): We incorporated the 2×2 head-swap experiments and updated the ObjectNet results. Bootstrap confidence intervals and statistical analyses are provided in Appendix F.
  3. SS2D memory analysis (Sec. 5.1): We added memory-length restriction experiments. Results across different memory-length bounds are provided in Appendix D, and an evaluation using input-dependent step sizes is included in Appendix C.4.
  4. Additional revisions: We clarified the configuration of the SS2D-removed model in Appendix A and revised the Abstract, Introduction, and Discussion to reflect the additional findings.

All revisions and additions are highlighted in blue. We thank the reviewers for their constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.