- Rating
- 4(+1)
- Confidence
- 3(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether a state-space model, specifically VMamba, can serve as the vision encoder in an autoregressive VLM. Under a shared downstream VLM recipe, the authors compare VMamba with ViT and Swin across contrastive pretraining, ImageNet classification, COCO detection adaptation, and ADE20K segmentation adaptation. The results show that VMamba generally achieves higher average performance, with the clearest gains on referring-expression grounding and smaller differences on VQA. The paper also finds that adding LayerNorm before the connector can substantially improve performance, whereas changing the MLP connector configuration produces smaller and less consistent gains.
Critical strengths
-
The problem is practically relevant. Most current VLMs still use ViT-based vision encoders, so determining whether SSMs can provide a viable alternative is valuable. The paper also shows that the vision-language interface can substantially affect encoder comparisons, which is important for future work.
-
The main empirical pattern is reasonably consistent. The authors do not rely on a single pair of CC12M checkpoints, but repeat the comparison under classification, detection, and segmentation settings. Tables 1-4 and Figure 3 show that VMamba's most consistent advantage occurs on grounding, and that this pattern can be observed across multiple model sizes and input resolutions.
-
The LayerNorm experiment in Section 6.1 is convincing. Figure 4 and Table 8 show that changing only the normalization before the connector can produce large performance differences. This suggests that an apparently weaker encoder may simply be poorly interfaced with the language model, which is a clear and useful insight from the paper.
-
The figures and tables are clear and easy to understand. Figure 1 provides an effective overview of the experimental design, Tables 1-4 separately summarize the VQA and grounding results, and Figures 3-4 clearly illustrate the effects of input resolution and interface choices. These presentation choices make the main trends across encoders and experimental settings easy to compare.
Critical weaknesses
- The current experiments do not fully isolate the effect of the encoder architecture. Section 3 uses the penultimate block of ViT but stage-3 features from VMamba and Swin. Although these features have the same number of tokens, they do not necessarily represent the same semantic depth. Appendix C compares stage-3 and stage-4 features only for VMamba and does not provide a corresponding ViT layer comparison. In addition, Section 4.1 states that the CC12M checkpoints differ in initialization, training schedule, and text encoder, while the ImageNet checkpoints used in Section 5.1 may also have different training recipes. I understand that the authors may primarily intend to compare practically available checkpoints rather than strictly demonstrate that the SSM architecture itself is superior. Nevertheless, the architecture-level claims should be stated more carefully, or supported by a ViT intermediate-layer sweep and a more closely matched pretraining comparison.
- The conclusion from the connector experiment should be narrowed. Appendix A shows that the two-layer and three-layer MLPs differ not only in depth but also in hidden width. The experiment therefore does not isolate the effect of connector depth. The phrase "increasing the depth" in the abstract should be replaced with the more general "changing the connector configuration," or supported by a parameter- or width-matched comparison.
Questions for authors
-
If the feature extraction layer for ViT and VMamba were selected using the same validation procedure, would VMamba retain its advantage on grounding?
-
Can the authors provide a parameter- or width-matched comparison of the two-layer and three-layer connectors? If not, I suggest revising the conclusion about connector depth to refer more generally to connector configuration.
We thank the reviewer for their constructive feedback and positive assessment of our work. We appreciate the suggestions regarding feature-layer selection, pretraining differences, and connector design. We conducted additional ablations in the updated version and clarified the scope of our conclusions below.
W1/Q1: Feature extraction layer selection. Thank you for pointing this out. We conducted additional feature-layer ablations showing that VMamba’s grounding advantage persists when comparing the best-performing layers among those evaluated, addressing the concern about feature-layer selection (Appendix C, Table 5).
For CC12M-pretrained ViT-B, blocks 9, 11, and 12 yield relatively comparable performance (46.7–48.4 overall; 40.1–43.5 grounding), with block 9 performing best. Nevertheless, VMamba-B retains its grounding advantage over this stronger ViT configuration (48.4 vs. 43.5). For ImageNet-1K-pretrained hierarchical backbones, stage 3 outperforms stage 4 on grounding in five of six comparisons. The exception is Swin-B, where stage 4 improves grounding from 20.2 to 22.9, but remains below VMamba-B (30.0). CC12M-pretrained VMamba-B also favors stage 3 (48.4 vs. 26.9).
Stage 3 provides more visual tokens than stage 4 (196 vs. 49), potentially preserving finer spatial information. However, feature depth also changes, so we cannot attribute the differences solely to token count. Overall, our main grounding observations remain robust across the evaluated feature taps.
W1: ImageNet pretraining differences. Thank you for the suggestion; as you pointed out, our work aims to compare practically available checkpoints, and our experiments aim to compare how these pretrained representations transfer to VLMs under a shared downstream setting, rather than strictly isolating architecture from pretraining. We also agree that the ImageNet-1K checkpoints may differ in their pretraining recipes. We select established, well-performing checkpoints from each architecture family to ensure that the comparison uses reasonably strong representations. We clarify the checkpoint-selection rationale and the scope of our conclusions in Sec. 5.1 and Appendix A.
W2/Q2: Connector depth and width. We agree that the original 2-layer and 3-layer MLPs differ in both depth and hidden width, so their performance differences cannot be attributed solely to depth. We additionally evaluate a 3-layer MLP-N with the same hidden width as the 2-layer MLP (we include these new results in Section 4.2 and Appendix F, Table 12). This configuration decreases overall performance for both ViT-B (47.4 → 43.8) and VMamba-B (50.7 → 49.9), showing that adding depth at fixed hidden width does not necessarily improve performance under our training recipe.
We also revised the abstract to use “changing the MLP connector configuration” instead of “increasing the depth,” avoiding attributing the observed effects solely to connector depth. Our original experiments remain informative for comparing complete connector configurations, including their effects on downstream performance and patch- and image-level probes (Section 6.2).