acceptodds
Under review as a conference paper at ICLR 2027

State-Space Models as Vision Encoders for VLMs: A Study across Training Settings and Interfaces

Abstract

Vision-Language Models (VLMs) typically leverage a vision encoder whose image features are mapped into a Large Language Model through a connector. While transformer-based vision encoders remain the standard choice, we ask whether state-space models (SSMs) can provide a viable alternative, given their competitive performance on vision tasks. To answer this question, we evaluate VMamba across four pre-trained setups (i.e., contrastive, classification, detection adaptation, and segmentation adaptation) and find that, under a shared downstream VLM recipe, the resulting models are generally competitive across VQA and referring-expression benchmarks, with the most consistent gains on grounding tasks. Moreover, we investigate the vision-language interface and find that (i) adding a normalization layer before the connector can substantially improve the VLM performance, and (ii) increasing the depth of an MLP connector produces inconsistent gains, while representation probes show that different connector depths alter the balance of local and global information in the visual tokens. Together, these results position VMamba as a competitive vision encoder alternative for VLMs, and highlight the importance of controlling the connector interface when comparing frozen vision encoders.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, vision encoders, state-space models, visual representations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_GtJaOpenReview ↗
Rating
4(+1)
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether a state-space model, specifically VMamba, can serve as the vision encoder in an autoregressive VLM. Under a shared downstream VLM recipe, the authors compare VMamba with ViT and Swin across contrastive pretraining, ImageNet classification, COCO detection adaptation, and ADE20K segmentation adaptation. The results show that VMamba generally achieves higher average performance, with the clearest gains on referring-expression grounding and smaller differences on VQA. The paper also finds that adding LayerNorm before the connector can substantially improve performance, whereas changing the MLP connector configuration produces smaller and less consistent gains.

Critical strengths
  1. The problem is practically relevant. Most current VLMs still use ViT-based vision encoders, so determining whether SSMs can provide a viable alternative is valuable. The paper also shows that the vision-language interface can substantially affect encoder comparisons, which is important for future work.

  2. The main empirical pattern is reasonably consistent. The authors do not rely on a single pair of CC12M checkpoints, but repeat the comparison under classification, detection, and segmentation settings. Tables 1-4 and Figure 3 show that VMamba's most consistent advantage occurs on grounding, and that this pattern can be observed across multiple model sizes and input resolutions.

  3. The LayerNorm experiment in Section 6.1 is convincing. Figure 4 and Table 8 show that changing only the normalization before the connector can produce large performance differences. This suggests that an apparently weaker encoder may simply be poorly interfaced with the language model, which is a clear and useful insight from the paper.

  4. The figures and tables are clear and easy to understand. Figure 1 provides an effective overview of the experimental design, Tables 1-4 separately summarize the VQA and grounding results, and Figures 3-4 clearly illustrate the effects of input resolution and interface choices. These presentation choices make the main trends across encoders and experimental settings easy to compare.

Critical weaknesses
  1. The current experiments do not fully isolate the effect of the encoder architecture. Section 3 uses the penultimate block of ViT but stage-3 features from VMamba and Swin. Although these features have the same number of tokens, they do not necessarily represent the same semantic depth. Appendix C compares stage-3 and stage-4 features only for VMamba and does not provide a corresponding ViT layer comparison. In addition, Section 4.1 states that the CC12M checkpoints differ in initialization, training schedule, and text encoder, while the ImageNet checkpoints used in Section 5.1 may also have different training recipes. I understand that the authors may primarily intend to compare practically available checkpoints rather than strictly demonstrate that the SSM architecture itself is superior. Nevertheless, the architecture-level claims should be stated more carefully, or supported by a ViT intermediate-layer sweep and a more closely matched pretraining comparison.
  2. The conclusion from the connector experiment should be narrowed. Appendix A shows that the two-layer and three-layer MLPs differ not only in depth but also in hidden width. The experiment therefore does not isolate the effect of connector depth. The phrase "increasing the depth" in the abstract should be replaced with the more general "changing the connector configuration," or supported by a parameter- or width-matched comparison.
Questions for authors
  1. If the feature extraction layer for ViT and VMamba were selected using the same validation procedure, would VMamba retain its advantage on grounding?

  2. Can the authors provide a parameter- or width-matched comparison of the two-layer and three-layer connectors? If not, I suggest revising the conclusion about connector depth to refer more generally to connector configuration.

1 comment

We thank the reviewer for their constructive feedback and positive assessment of our work. We appreciate the suggestions regarding feature-layer selection, pretraining differences, and connector design. We conducted additional ablations in the updated version and clarified the scope of our conclusions below.

W1/Q1: Feature extraction layer selection. Thank you for pointing this out. We conducted additional feature-layer ablations showing that VMamba’s grounding advantage persists when comparing the best-performing layers among those evaluated, addressing the concern about feature-layer selection (Appendix C, Table 5).

For CC12M-pretrained ViT-B, blocks 9, 11, and 12 yield relatively comparable performance (46.7–48.4 overall; 40.1–43.5 grounding), with block 9 performing best. Nevertheless, VMamba-B retains its grounding advantage over this stronger ViT configuration (48.4 vs. 43.5). For ImageNet-1K-pretrained hierarchical backbones, stage 3 outperforms stage 4 on grounding in five of six comparisons. The exception is Swin-B, where stage 4 improves grounding from 20.2 to 22.9, but remains below VMamba-B (30.0). CC12M-pretrained VMamba-B also favors stage 3 (48.4 vs. 26.9).

Stage 3 provides more visual tokens than stage 4 (196 vs. 49), potentially preserving finer spatial information. However, feature depth also changes, so we cannot attribute the differences solely to token count. Overall, our main grounding observations remain robust across the evaluated feature taps.

EncoderPretrainingTapTokensOverall AvgGrounding Avg
Swin-BIN1KStage 319637.620.2
Stage 44939.022.9
VMamba-BCC12MStage 319650.748.4
Stage 44941.626.9
ViT-BCC12MBlock 919648.443.5
Block 11 (default)19647.441.3
Block 1219646.740.1

W1: ImageNet pretraining differences. Thank you for the suggestion; as you pointed out, our work aims to compare practically available checkpoints, and our experiments aim to compare how these pretrained representations transfer to VLMs under a shared downstream setting, rather than strictly isolating architecture from pretraining. We also agree that the ImageNet-1K checkpoints may differ in their pretraining recipes. We select established, well-performing checkpoints from each architecture family to ensure that the comparison uses reasonably strong representations. We clarify the checkpoint-selection rationale and the scope of our conclusions in Sec. 5.1 and Appendix A.


W2/Q2: Connector depth and width. We agree that the original 2-layer and 3-layer MLPs differ in both depth and hidden width, so their performance differences cannot be attributed solely to depth. We additionally evaluate a 3-layer MLP-N with the same hidden width as the 2-layer MLP (we include these new results in Section 4.2 and Appendix F, Table 12). This configuration decreases overall performance for both ViT-B (47.4 → 43.8) and VMamba-B (50.7 → 49.9), showing that adding depth at fixed hidden width does not necessarily improve performance under our training recipe.

We also revised the abstract to use “changing the MLP connector configuration” instead of “increasing the depth,” avoiding attributing the observed effects solely to connector depth. Our original experiments remain informative for comparing complete connector configurations, including their effects on downstream performance and patch- and image-level probes (Section 6.2).

ConnectorViT-B/16VMamba-B
2-layer MLP47.450.7
3-layer MLP-N (width-matched)43.849.9
3-layer MLP (original)46.949.5
Reviewer_NAr8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the vision encoder component of standard autoregressive VLM design that consists of a vision encoder, connector, and a decoder-only language model. The main goal is to understand the effectiveness and viability of state-space vision backbones such as VMamba in such a VLM design, by comparing them against the currently adopted ViT and Swin encoders. They used a shared VLM recipe and investigate different training objectives for vision encoder training such as contrastive learning, classification, detection, and segmentation. The evaluations are performed on VQA and grounding benchmarks.

Critical strengths

The paper puts a reasonable effort into the fairness of experimental setup to investigate different vision encoder choices. Although currently being gradually replaced by encoder-free native multimodal architectures, encoder based VLMs are still relevant and practical, and this paper takes another look into this recipe using state-space models.

Critical weaknesses

My main criticism for the paper is the lack of actionable new findings that resulted from it:

  1. Empirical findings such as adding normalization layer before the connector or increasing the connector depth is not new. In fact, for many VLM practitioners, these ablations are quite natural and well-investigated.
  2. While the findings are not new, the paper could still have provided insights for VLM practitioners. However, the results are both quite sub-par in terms of VLM performance, which makes it hard to draw any practical conclusions for the current landscape we have, and they are also quite narrow in terms of the capabilities they are measuring. Ideally, for this study to be relevant to practitioners, baselines should be much stronger with more recent training data and recipes, and evaluated on a broad set of downstream VLM tasks including VQA, captioning, reasoning, instruction following, grounding.
Questions for authors

None, I believe the current experimental setup together with the baselines and evaluation suite is not sufficient to draw strong conclusions, thus I recommend a major revision and resubmission.

1 comment

We thank the reviewer for their constructive feedback. Our main contribution is an empirical study of how pretrained vision encoders transfer to VLMs and how sensitive they are to the connector interface. We do not claim or aim to introduce a new VLM component or achieve state-of-the-art performance on standard benchmarks. We elaborate on these points below.


W1: Novelty and significance of the empirical findings. We agree that LayerNorm and MLP connectors are standard techniques, and we do not claim otherwise. Our contribution is focused on demonstrating that interface choices can fundamentally change conclusions about vision-encoder effectiveness, particularly when comparing encoders across architecture families. As an example, we show in Table 11 that without LayerNorm, ViT-B outperforms VMamba-B by 10.3 points overall (46.6 vs. 36.3); while with LayerNorm, the ranking reverses, and VMamba-B leads by 3.3 points (50.7 vs. 47.4). Moreover, LayerNorm improves all 17 tested configurations in Fig. 4.

Our study also investigates how vision encoders transfer across four training setups, and our focus is to provide concrete empirical evidence about how encoder architecture, training objectives, and the vision-language connector interact. For example, when adapted using the same COCO detection and ADE20K segmentation setups, VMamba-S outperforms ViT-S on grounding by 5.9 points (59.0 vs. 53.1) and 6.8 points (49.0 vs. 42.2), respectively (Tables 3-4). Moreover, we also investigate why larger ImageNet-pretrained encoders show worse grounding performance, and we probe how trained connectors change the accessibility of local and global visual semantics (Fig. 5). Together, these findings provide empirical evidence about the interaction between vision-encoder architecture, training objectives, and the vision-language interface.


W2: VLM recipe and evaluation scope. Our training setup is designed to investigate performance differences across frozen vision encoders from different families, while also testing multiple language backbones when choosing that setup (Fig. 2).

Following prior work [1, 2], we indeed focus on evaluating vision encoders in tasks that emphasize visual perception and localization across 13 benchmarks covering VQA, counting, spatial understanding, and referring-expression grounding.

Our findings suggest that other visual backbones beyond ViT show great potential, and suggest that SSM encoders are a competitive alternative in VLMs for tasks that require stronger visual perception and localization capabilities. Our work also investigates how different vision encoder architectures perform when adapted under identical downstream conditions, and highlights the importance of carefully examining interface choices in VLMs, as they can impact the effectiveness of each component and overall performance.


References:

[1] Tong, Shengbang, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." Advances in Neural Information Processing Systems 37 (2024): 87310-87356.

[2] Chen, Lin, et al. "Are we on the right way for evaluating large vision-language models?." Advances in Neural Information Processing Systems 37 (2024): 27056-27087.

Reviewer_BEniOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies VMamba as a frozen vision encoder for autoregressive vision-language models. Under a shared downstream training recipe, the authors compare VMamba with Transformer encoders across four settings: contrastive pretraining, image classification, detection adaptation, and segmentation adaptation. They also examine pre-connector LayerNorm, MLP connector configurations, and visual representations before and after the connector. The results show that VMamba’s main advantages lie in referring-expression grounding, with smaller gains on visual question answering. Normalization substantially improves performance for some configurations.

Critical strengths
  1. Broad experimental coverage. The study covers four encoder-training settings and 13 benchmark scores, matching the downstream recipe, input resolution, and visual token count within each setting. The more uniform detection and segmentation adaptation settings provide useful evidence for comparing visual encoders.

  2. Meaningful grounding improvements. In Table 1, CC12M-pretrained VMamba-B achieves an average grounding score of 48.4, compared with 41.3 for ViT-B, and leads on all five grounding evaluations. Similar trends appear after detection and segmentation adaptation.

  3. Practical insights into normalization. In Table 8, adding LayerNorm improves the overall score of CC12M-pretrained VMamba-B from 36.3 to 50.7. This demonstrates that interface design can substantially affect the downstream performance of frozen visual encoders.

Critical weaknesses
  1. The incremental contribution is unclear. The prior work Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders already covers a similar research framework and several central findings. This paper does not discuss that work or sufficiently explain what new insights its additional experiments establish.

  2. The conclusions have limited scope. The experiments cover only VMamba, without comparisons to other SSM architectures or hybrid backbones such as MambaVision. The contrastive experiments are also limited to CC12M, leaving performance under larger-scale pretraining unclear.

Questions for authors
  1. Compared with arXiv:2603.19209, what is the paper’s main new finding? Which additional experiments change an existing conclusion or provide new design guidance?

  2. Why can a three-layer connector reduce linear-probe performance while improving downstream grounding? What explanation do the existing results support?

2 comments

We thank the reviewer for the positive assessment and constructive feedback. In particular, the points about backbone coverage and linear probing motivate new analysis and results that we incorporate in the updated version. We address each concern in detail below.

W1 / Q1: We thank the reviewer for pointing out the related preprint. While both works share the main research question, this work makes several substantive extensions and contributes more controlled experimental comparisons, providing additional insights into the vision-language interface.

  1. Better controlled comparisons in dense tasks. The earlier comparisons in dense tasks used off-the-shelf checkpoints with different training recipes, input resolutions, and visual-token counts, which confounds the architectural effect with many other factors. In this work, we adapt ViT, Swin, and VMamba using shared detection and segmentation training procedures with matched resolutions and token counts (Section 5.2, Tables 3-4). We further repeat the comparisons with checkpoints separately adapted at 224, 256, and 512 resolutions (Fig. 3). These experiments provide stronger evidence that VMamba's grounding advantages are not specific to particular checkpoint configurations.

  2. Systematic validation of the VLM interface. Using the same downstream recipe does not guarantee a fair comparison, since different encoders may respond differently to the same interface. We therefore investigate the language backbone, along with normalization and connector choices (Fig. 2), and further validate the selected interface across all four settings for training the encoder (Section 6.1, Fig. 4). In particular, pre-connector LayerNorm improves performance in all 17 tested configurations, demonstrating the importance of controlling factors related to the interface when comparing frozen encoders.

  3. Probing how connectors reshape visual information. We also investigate how connectors transform visual representations after training, which goes beyond the observations that connector changes can affect downstream performance. Section 6.2 probes patch-level and image-level semantic information before and after the connector, revealing different changes in local and global information across connector configurations (Fig. 5). This provides additional insight into the role of the vision-language interface beyond conventional backbone-level probing (Sections 6.2 and D.2).

Our evaluation also includes contrastively pretrained VMamba and additionally evaluates on CV-Bench 2D and Flickr30k Entities. Together, these experiments provide stronger evidence under controlled comparisons and more actionable guidance for evaluating SSM-based vision encoders in VLMs.

W2: Backbone coverage and pretraining scale. We focus on representative Transformer (ViT, Swin) and SSM (VMamba) backbones to compare their transfer behavior under a shared VLM recipe. To broaden architectural coverage, we included MambaVision-T/S/B (a hybrid Mamba-Transformer) in the ImageNet-1K classification setting (results available in the updated version: Appendix E, Table 8). These models achieve 47.4/49.0/49.5 overall and 43.0/45.4/46.2 grounding, respectively, demonstrating competitive VLM performance beyond VMamba.

These results provide encouraging evidence for another SSM-containing architecture, although the main conclusions remain specific to the evaluated backbones and training settings. In particular, the hybrid results do not isolate the contribution of SSM components, and transfer to larger-scale contrastive pretraining remains an open question.

Visual EncoderOverall AvgGrounding Avg
MambaVision-T47.443.0
MambaVision-S49.045.4
MambaVision-B49.546.2
Swin-T44.836.6
Swin-S41.527.8
Swin-B37.620.2
ViT-S/1643.033.2
ViT-B/1643.533.2
VMamba-T47.842.9
VMamba-S47.642.1
VMamba-B42.630.0

Q2: Why can a three-layer connector reduce linear-probe performance while improving grounding? Linear probes measure how readily object-category information can be decoded from visual tokens, whereas grounding evaluates how effectively the VLM uses those tokens together with language to localize a region. Because the connector and LLM are jointly optimized for downstream tasks, not for linear-probe performance, lower probe recall does not necessarily imply worse grounding.

We include additional analysis in Appendix D.2 (Table 7), showing that the 3-layer MLP reduces both patch-level (RpR_p) and image-level (RiR_i) recall in all 10 detection and segmentation comparisons, yet improves grounding in 8. Notably, Ri decreases more than RpR_p in every comparison, increasing Rp−RiR_p-R_i. This suggests that the relative accessibility of local versus global category information may be more relevant than absolute probe recall alone, although the gap does not directly measure language-region alignment or establish causality.

Reviewer_KW5gOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents an empirical study evaluating State-Space Models (specifically VMamba) as frozen vision encoders for autoregressive VLMs, comparing them against standard ViT and Swin Transformer baselines. To keep the comparison fair, the authors lock in a shared downstream VLM training recipe (Vicuna-7B, 2-layer MLP connector, LLaVA-1.5 665K instruction tuning data) and evaluate the encoders across four pretraining setups: CC12M contrastive learning, ImageNet-1K classification, COCO object detection adaptation, and ADE20K segmentation adaptation. Across 8 VQA and 5 referring-expression grounding benchmarks, VMamba generally matches or outperforms ViT, showing the most consistent gains on grounding tasks. The authors also examine the vision-language interface, demonstrating that applying a frozen LayerNorm right before the connector heavily boosts VLM performance, and use linear probes to analyze how 2-layer versus 3-layer MLP connectors shift the balance between local patch-level and global image-level information.

Critical strengths

1- controlled multi-setting evaluation and interface insight: comparing vision backbones in VLMs is usually messy because public checkpoints vary wildly in pretraining data and downstream recipes. By isolating four distinct pretraining regimes and adapting the detection and segmentation backbones using the exact same training code, the authors provide a clean, apples-to-apples comparison between SSMs and Transformers. furthermore, their finding on pre-connector LayerNorm is a very practical takeaway, showing that without it, certain frozen encoders (like contrastive VMamba-B or classification ViTs) degrade severely on grounding, which could easily lead researchers to draw the wrong conclusions when benchmarking new vision backbones.

2- strong empirical showing for SSMs in spatial grounding: the paper clearly establishes that VMamba is not just viable for VLMs, but actually beats plain ViT and hierarchical Swin transformers consistently on referring-expression grounding across different resolutions (2242224^2 to 5122512^2), while offering much better latency scaling at extreme resolutions.

Critical weaknesses

-- confounded contrastive comparison and lack of web-scale validation: in the contrastive setup (which is the most standard paradigm for modern VLMs), the comparison between CLIP ViT-B and CLIMP VMamba-B on CC12M is not strictly controlled. As the authors admit, VMamba-B was initialized from ImageNet-1K weights while ViT-B was trained from scratch, and they used different text encoders and schedules. Giving VMamba an ImageNet initialization advantage on a relatively small 12M dataset is a major confounder, and there is no experiment showing if VMamba's edge holds up at true web scale (e.g., 400M+ image-text pairs) where standard CLIP/SigLIP ViTs dominate.

--unexplained inverse scaling on ImageNet and shallow efficiency profiling: in table 2 (ImageNet-1K classification), VLM performance actually drops quite a bit as the backbone scales up from Small to Base for both VMamba-B (47.6 down to 42.6) and Swin-B (44.8 for Swin-T down to 37.6 for Swin-B), primarily due to a collapse in grounding scores. The paper notes that ImageNet top-1 accuracy doesn't predict VLM performance, but doesn't really dig into why larger models fail here. Also, the latency profiling in figure 7 only measures the isolated vision encoder up to 819228192^2; in a real VLM, feeding thousands of visual tokens into a 7B LLM makes the LLM prefill the main bottleneck, so encoder-only timing is a bit misleading without end-to-end TTFT (time-to-first-token) numbers.

Questions for authors

Clarification: In table 2, why do VMamba-B and Swin-B suffer such a sharp drop in grounding performance compared to their Tiny and Small variants when using ImageNet-1K classification weights, especially since this drop disappears after COCO/ADE20K adaptation or when fine-tuned at 2562256^2/5122512^2 (figure 3)? Is there a specific feature norm or representation collapse issue in the deeper stage-3 layers of the Base models at 2242224^2?

  • Clarification: In figure 2 (right) and table 9, the 6-layer MLP and attention-based connectors (Cross-Attn and Self-Attn) perform terribly, dropping overall scores by ~15 points. Were hyperparameters (like learning rate or warmup) tuned at all for these deeper/attention connectors, or did they simply fail to converge under the default 1-epoch schedule tuned for the 2-layer MLP?
2 comments

We thank the reviewer for the positive assessment and constructive feedback. In particular, the questions about inverse scaling and end-to-end latency motivate two new analyses that we incorporate in the updated version. Below, we clarify the scope of the comparison, present additional evidence, and address connector optimization.


W1: Contrastive pretraining confounds and web-scale generalization. We agree that the CC12M comparison does not strictly isolate architecture. VMamba-B uses ImageNet-1K initialization, while the checkpoints also differ in text encoders and training schedules. Our primary purpose in using this pair is to establish a strong shared downstream VLM recipe; thus, we interpret this experiment as a comparison of practically available contrastive encoders under matched downstream VLM training. We further investigate the selected interface across all four encoder-training settings (Fig. 4). More importantly, VMamba's grounding advantage also appears in the better-controlled COCO detection and ADE20K segmentation adaptation experiments (Tables 3-4), which use shared adaptation procedures. Web-scale contrastive pretraining for SSM vision encoders remains underexplored; whether these advantages persist at larger scales is an open question.


W2 / Q1: Inverse scaling of ImageNet pretrained encoders. One possible explanation is that ImageNet classification emphasizes recognizing image-level object categories, often associated with prominent objects, whereas grounding requires localizing a region specified by textual descriptions, which may not be the most prominent object in the image. We include additional probes in Appendix D.2, Table 6, which shows that as the VMamba and Swin pretrained models scale from Tiny to Base, both patch-level (RpR_p) and image-level (RiR_i) recall increase, but RiR_i increases faster. Consequently, Rp−RiR_p-R_i decreases (−11.7 → −20.4 for VMamba; −4.6 → −12.5 for Swin), alongside grounding declines (42.9 → 30.0; 36.6 → 20.2). This pattern is not consistently observed after detection or segmentation adaptation.

Regarding feature norms, we additionally measure the RMS of stage-3 and stage-4 encoder features (table below). RMS accounts for differences in feature dimension and generally increases with model size, but does not consistently explain grounding differences across feature taps. Moreover, we apply LayerNorm before the connector, reducing sensitivity to raw feature magnitude. To investigate whether there is representation collapse, we apply the linear probing protocol from CLIP [1] to evaluate features from stages 3 and 4. We observe no performance degradation at stage 3, providing additional evidence against representation collapse as the primary explanation. Nevertheless, these results do not rule out more localized or task-specific representation degradation.

Mean per-token RMS before LayerNorm (COCO val2017, 224 × 224):

EncoderStage 3 RMSStage 4 RMS
Swin-T3.113.80
Swin-S6.784.27
Swin-B9.868.21
VMamba-T1.302.13
VMamba-S2.743.85
VMamba-B3.444.70

For N visual tokens with feature dimension D, RMS is computed as 1N∑t=1N∥xt∥2D\frac{1}{N}\sum_{t=1}^{N}\frac{\|x_t\|_2}{\sqrt{D}}, then averaged across images.

Linear probing of ImageNet-1K pretrained encoders: Stage 3 vs. Stage 4

EncoderStage 3Stage 4
Swin-B77.7276.06
VMamba-B79.5477.94

W2: Latency Profiling. We agree that processing thousands of visual tokens makes LLM prefill expensive. To quantify the impact, we extend Fig. 7 with encoder-to-LLM prefill-path measurements using Vicuna-7B and Gemma-2-2B, without token compression. At 819228192^{2}, VMamba-B reduces encoder-only latency by 76.8%, translating to 10.1% lower prefill-path latency with Vicuna-7B (68.56s → 61.63s) and 27.7% with Gemma-2-2B (24.75s → 17.88s). These results show that encoder efficiency matters more when language-side computation is less expensive. We include these results in Section 7 and Fig. 7 accordingly.


References:

[1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PmLR, 2021.

Q2: Optimization of complex connectors. We agree that complex connectors may require different optimization settings. To clarify, we use the same one-epoch instruction-tuning recipe for all connectors (learning rate 210-5, 3% warmup) to maintain a shared comparison setting, without connector-specific tuning (Appendix A, Section 4.2).

To examine whether additional training helps, we extend cross-attention training to two epochs. Overall performance improves for both encoders, but remains below the 2-layer MLP baseline (47.4 for ViT-B; 50.7 for VMamba-B).

EncoderCross-attn (1 epoch)Cross-attn (2 epochs)
ViT-B35.738.0
VMamba-B36.238.0

These results show that longer training helps, but two epochs do not close the performance gap. As a result, we don't view these scores as proof that the architecture is fundamentally weaker or incapable of converging.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.