acceptodds
Under review as a conference paper at ICLR 2027

Deep and Compact Visual Experts for Fast Encoder-Free MLLM

Abstract

Encoder-free Multimodal Large Language Models (MLLMs) process images and text within a single backbone, enabling early cross-modal fusion and eliminating the prefill latency incurred by vision encoders in modular MLLMs like LLaVA. This prefill advantage is particularly important for edge deployment and real-time interactions. Despite this advantage, most encoder-free architectures ignore the latency aspect. In this paper, we first examine the design space of encoder-free architectures to measure both multimodal performance and latency. Our analysis shows that existing encoder-free designs are limited by insufficient visual processing depth in shallow backbones. Building on this finding, we propose DeCoVE, a Deep and Compact Visual Expert architecture that decouples visual processing depth from backbone depth by trading visual width for depth. We then demonstrate the performance–latency tradeoff of DeCoVE by varying its depth. Moreover, DeCoVE can be initialized from parts of pretrained vision encoders, bringing together the training efficiency of pretrained models and the prefill advantage of encoder-free designs. Based on DeCoVE, we develop DeCoVE-3B, an encoder-free MLLM that achieves state-of-the-art performance among encoder-free baselines and is competitive with Qwen3.5-VL 2B, while reducing prefill latency by 41.5% across 13 benchmarks. DeCoVE-3B establishes a new Pareto frontier between performance and prefill latency for MLLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision, Vision Language Model, Multimodal Large Language Model, Encoder Free, Efficient VLM, MLLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5XcgOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies encoder-free multimodal language models under latency-constrained settings. The authors argue that small encoder-free backbones suffer from insufficient visual processing depth and propose DeCoVE, a deep and narrow visual expert that operates in parallel with the text backbone. DeCoVE supports early cross-modal fusion and initialization from pretrained vision-transformer weights. The resulting DeCoVE-3B model is evaluated on 13 visual understanding, document understanding, and grounding benchmarks. It outperforms existing encoder-free baselines and reports substantially lower time-to-first-token than several modular MLLMs. Ablations examine visual depth, layer placement, early fusion, and pretrained initialization.

Critical strengths
  1. Practical and well-motivated architecture. DeCoVE addresses an important limitation of small encoder-free MLLMs by increasing visual depth while retaining parallel image–text processing.

  2. Strong empirical evaluation. The paper evaluates 13 benchmarks and provides useful ablations on visual depth, layer placement, early fusion, and pretrained initialization. The reported accuracy–latency trade-off is compelling.

Critical weaknesses
  1. Some central implementation details appear inconsistent. Section 4.1 specifies sequential 16x1616x16 and 32x3232x32 convolutions, implying an effective stride of 512, which seems incompatible with the reported visual-token counts. Section 5.1 also describes 4x10−84x10^-8 as a higher learning rate than 5x10−55x10^-5. These specifications need correction before the model can be reproduced.

  2. The latency and training-data protocols are insufficiently documented. Since efficiency is a central claim, the paper should report batch size, precision, warm-up runs, repetitions, variance, actual token lengths, and the components included in TTFT. The pretraining and instruction-tuning datasets are described only by broad categories, making contamination and controlled comparison difficult to assess.

Questions for authors
  1. What are the exact kernel sizes and strides of the two image-embedding convolutions? How do they produce up to 2,560 visual tokens?

  2. Is the learning rate for the newly initialized layers really 4x10−84x10^-8, or is this a typographical error?

  3. Could the authors provide the complete TTFT protocol, including batch size, precision, warm-up, number of runs, variance, and whether preprocessing is included?

  4. What datasets constitute the 3.8M VQA, 1.4M OCR, and 15M instruction-tuning samples? Were evaluation-set overlap and contamination checked?

  5. Given that DeCoVE contains a 24-layer vision-specific pathway initialized from Qwen-ViT, how do the authors distinguish it conceptually from a parallel or interleaved modular vision encoder?

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer 5Xcg

We thank the reviewer for the valuable feedback. We appreciate that the reviewer found our architecture "practical and well-motivated" with "strong empirical evaluation", and that the "accuracy–latency trade-off is compelling". We respond to the issues and questions below.

W1. Inconsistent implementation details on convolution

Thank you for catching this. The text in Section 4.1 was miswritten. Conv1 uses a 16×1616 \times 16 kernel with stride 16, and Conv2 uses a 2×22 \times 2 kernel with stride 2. The total effective stride is therefore 32, and each visual token covers a 32×3232 \times 32 pixel region. A budget of 2,560 visual tokens corresponds to images of about 2.6 MP; for example, a 1280×20481280 \times 2048 image gives a 40×6440 \times 64 grid of 2,560 tokens. For images that exceed this 2,560-token cap, we first resize the image so that it fits within the budget. We have corrected Section 4.1 in the revised PDF.

Q2. Is the 4×10−84 \times 10^{-8} learning rate a typo?

Yes, it is a typo. The correct learning rate is 8×10−48 \times 10^{-4}. We have corrected it in the revised PDF.

Q3. Complete TTFT protocol

The TTFT protocol uses a batch size of 1 and BF16 precision. For warm-up, we run 3 untimed generations prior to profiling. Then, for each sample, we run 1 untimed generation before the 3 timed prefills, so that the kernels for the input image's shape are ready. We profile all samples of each evaluation, running the prefill 3 times per sample; we take the median of the 3 runs for each sample and then average over the samples. The standard error of the mean is 0.53 ms for DeCoVE-3B (26.16±0.5326.16 \pm 0.53 ms), compared with 44.63±1.4444.63 \pm 1.44 ms for Qwen3.5 2B. Image preprocessing and text tokenization are excluded from the profiling. We have added these details in Appendix D.1.

Q4. Composition of the VQA, OCR and instruction-tuning data; contamination checks

The 3.8M VQA data for pretraining are sampled from LLaVA-665k [1], Molmo2 [2], PixMo-Points [3], and MGrounding-630k [4]. The 1.4M OCR data are mostly from FinePDF document data and in-house rendered text datasets for natural-image OCR capabilities. The 15M instruction tuning data are sampled from various publicly available sources; notable sources include LLaVA-OneVision [5], Cambrian [6], and Molmo2 [2]. For all of our training data, we perform exact deduplication against the evaluation sets to remove contamination. We will release details on the data composition after paper acceptance and internal review.

Q5. Conceptual difference from a parallel or interleaved modular vision encoder

Thank you for raising this question. DeCoVE can indeed be viewed as a form of parallel or interleaved modular vision encoder. Our contribution is not to introduce a fundamentally separate category, but to study and refine this broader design space in the context of visual experts in encoder-free MLLMs. Specifically, we address four design questions:

  1. Why integrate a vision encoder in parallel with the language backbone? This design avoids placing the entire vision encoder before the language backbone, helping reduce prefill latency while retaining pretrained visual processing capabilities.
  2. How should the vision layers be arranged? A straightforward design pairs the 24 vision layers with 24 language backbone layers. However, Figure 5 shows that this arrangement is suboptimal because it reduces the total visual processing depth relative to a conventional modular MLLM. DeCoVE instead packs the vision layers asymmetrically into the early layers of the language backbone, allowing subsequent language layers to further process the visual representations and thereby increasing the total visual processing depth.
  3. How much of the pretrained vision encoder should be retained? Using the full vision encoder does not yield the best balance between performance and latency. Our ablations show that retaining 24 layers achieves the best performance, while retaining 18 layers offers the best latency–performance trade-off.
  4. When should visual and language representations interact? Rather than exposing the language backbone only to the final visual features, as in the cross-attention MLLM design of [7], DeCoVE enables early fusion between the two modalities. Table 5 demonstrates the benefit of this choice.

Thus, DeCoVE presents a specific design for a parallel vision encoder, motivated by the advantages and disadvantages of encoder-free MLLMs.

[1] Liu et al. LLaVA-1.5. CVPR 2024.

[2] Clark et al. Molmo2. CVPR 2026.

[3] Deitke et al. Molmo and PixMo. CVPR 2025.

[4] Li et al. Migician. ACL Findings 2025.

[5] Li et al. LLaVA-OneVision. TMLR 2025.

[6] Tong et al. Cambrian-1. NeurIPS 2024.

[7] Alayrac et al. Flamingo. NeurIPS 2022.

Reviewer_5n93OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the trade-off between visual understanding and prefill latency in small-scale encoder-free MLLMs. The authors argue that directly removing the vision encoder leaves small language backbones with insufficient depth for visual processing, limiting model performance.

To address this issue, the paper proposes DeCoVE, which introduces deeper, narrower visual experts into the early layers of the LLM backbone and enables cross-modal fusion through projections and the residual stream at global attention layers. This design also allows initialization from parts of a pretrained vision encoder to improve training efficiency.

The final model, DeCoVE-3B, is built on LFM2.5-2.6B and distributes the first 24 layers of Qwen-ViT across the first eight layers of the language backbone. The authors evaluate the model on 13 benchmarks and analyze the effects of visual depth, layer placement, initialization, and early fusion. The results show that DeCoVE achieves strong average performance among the evaluated encoder-free baselines while substantially reducing TTFT.

Critical strengths
  1. The paper proposes a practically valuable trade-off between performance and latency for small encoder-free MLLMs, achieving state-of-the-art performance among encoder-free models. Using a deeper, narrower visual branch and reusing pretrained visual weights constitutes a clear and practical architectural design.

  2. The authors provide valuable design analyses and ablation studies. Figure 2 analyzes the sources of the prefill latency advantage, Table 1 demonstrates the architectural benefits, and Figure 6 shows the advantages of pretrained visual initialization in both convergence and final multimodal performance.

Critical weaknesses

The paper primarily demonstrates its efficiency advantage through TTFT, but its practical impact on end-to-end response latency remains unclear. In Table 2, the absolute reductions relative to LFM2.5-VL 3B and Qwen3.5-VL 2B are 7.6 ms and 18.4 ms, respectively. If subsequent token decoding speeds are similar, these savings will not accumulate with output length and may account for only a small fraction of total latency in tasks requiring long responses.

Questions for authors
  1. Do DeCoVE’s gains over the original visual expert design persist as the language backbone becomes deeper? The paper attributes the performance bottleneck of small encoder-free models to insufficient visual processing depth, but this bottleneck may diminish as the backbone itself becomes deeper. Could the authors provide comparisons across different backbone depths to clarify the design’s applicability and whether its gains decrease with increasing backbone depth?

  2. Does the model’s text-only performance degrade relative to the original language backbone after multimodal pretraining and instruction tuning? Since the language backbone parameters are also updated during training, comparisons before and after training on text-only understanding, knowledge, and reasoning tasks would help establish whether the proposed training scheme preserves the original language capabilities while acquiring visual capabilities.

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer 5n93

We thank the reviewer for the thoughtful and detailed review. We are grateful that the reviewer found our architecture to be a "clear and practical architectural design", and that our work "provide valuable design analyses". Below, we address the questions and concerns of the reviewer.

W1. TTFT savings versus end-to-end latency for long responses

We agree that if the prefill latency is a small fraction of the total latency, the end-to-end latency difference with modular MLLMs (e.g., Qwen3.5 2B) would be small. However, for tasks that are dominated by prefill latency, such as short VQA on long videos or predicting agentic actions from high-resolution screenshots, DeCoVE will have significantly faster prefill and end-to-end latency. In the table below, we fine-tune DeCoVE on video instruction tuning data and evaluate on video understanding benchmarks. As shown in the table, DeCoVE is on average 103.2 ms faster in end-to-end latency compared to Qwen3.5 2B while offering competitive performance.

BenchmarkDeCoVE acc.Qwen3.5-2B acc.DeCoVE E2E (ms)Qwen3.5-2B E2E (ms)
ActivityNet-QA66.4754.70601.4683.8
MMVU39.8046.70516.5611.8
MotionBench60.3251.93491.2616.9
MVBench56.1557.33476.7585.9
Average55.6952.67521.4624.6

Q1. Do the gains persist with deeper language backbones?

We believe that as the language backbone gets deeper, the gains over visual expert designs will decrease, and we agree it is important to establish the design's applicability in terms of backbone size. However, this ablation is currently not feasible, as it would require pretrained language backbones of varying depth and size with the same architecture, which we do not have access to. We do think that as long as the language backbone is not significantly deeper than the vision encoder (e.g., 27 layers for SigLIP), DeCoVE would improve performance over visual experts.

On the other hand, we would like to highlight that the gains in terms of training token efficiency should remain as the language backbone gets deeper. Regardless of the depth, a visual expert initialized from scratch will converge slower than DeCoVE initialized from parts of a vision encoder. As shown in Figure 6 of the paper, DeCoVE trained for 50B tokens outperforms a visual expert trained for 300B tokens.

Q2. Text-only performance after multimodal training

We agree that retaining text performance after multimodal pretraining and instruction tuning is important. We would like to first clarify that most MLLMs update the language backbone parameters, and that text capability forgetting is a major problem for MLLMs in general [1].

Below, we compare the text performance of the checkpoints before multimodal pretraining, after multimodal pretraining, and after multimodal instruction tuning. Note that multimodal pretraining includes 30% text-only data, while multimodal instruction tuning has no text-only data. As shown in the table, the multimodal pretrained checkpoint performs on par with the pre-multimodal checkpoint on reasoning and knowledge tasks, while being better on instruction following. On the other hand, the multimodal instruction-tuned checkpoint performs worse. This demonstrates that text degradation is related to the data distribution, as discussed in [1], rather than caused by a particular architecture like DeCoVE. We have updated this in Appendix C.8

BenchmarkPre-MultimodalMultimodal PretrainSFT
MMLU57.657.752.0
WinoGrande62.064.654.0
IFEval48.858.457.9

[1] Lin, Ji et al. "VILA: On Pre-training for Visual Language Models." IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024).

Reviewer_KN4sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

DeCoVE addresses the key bottleneck of insufficient visual processing depth in encoder-free multimodal large language models (MLLMs) built on small backbones, by proposing a Deep and Compact Visual Expert architecture. The core idea is to trade visual expert width for depth while keeping the total parameter budget unchanged, thereby enhancing visual representation capacity without sacrificing parameter efficiency. DeCoVE can be initialized directly from partial weights of a pretrained vision encoder, combining training efficiency with inference speed. DeCoVE-3B, built on this architecture, surpasses all encoder-free baselines across 13 benchmarks and achieves performance comparable to the modular Qwen3.5-VL 2B, while reducing prefill latency by 41.5%. It establishes a new Pareto frontier between performance and latency, making it particularly well-suited for edge deployment and real-time interaction scenarios.

Critical strengths
  1. Rather than jumping straight to a new architecture, the paper first conducts a thorough design-space analysis to understand why existing encoder-free MLLMs fall short in small-backbone settings. The progressive transformation study — incrementally converting a modular MLLM into an encoder-free design while isolating the latency contribution of each change — is methodologically rigorous and rare. It yields a crisp, actionable diagnosis: insufficient visual processing depth is the primary bottleneck, not data scale or distillation quality. This finding alone shifts how researchers should think about the problem. The subsequent proposal of trading width for depth at constant parameter budget follows directly from this diagnosis, making the entire design rationale transparent and well-justified rather than ad-hoc. For a paper to both identify a hidden bottleneck and solve it cleanly within the same work is a strong signal of quality.

  2. DeCoVE is architecturally clean — no exotic modules, no auxiliary losses, no distillation dependency. It can be initialized directly from pretrained vision encoder weights, which dramatically lowers training cost and makes it immediately accessible to practitioners. Despite this simplicity, DeCoVE-3B achieves SOTA among all encoder-free baselines, matches the modular Qwen3.5-VL 2B in multimodal performance, and reduces prefill latency by 41.5%. Furthermore, the number and arrangement of visual layers provide a continuous knob for trading performance against speed, making the approach adaptable to diverse deployment constraints.

Critical weaknesses
  1. The paper positions edge deployment and real-time interaction as its core motivation, yet by late 2026, the industry has decisively shifted toward omni-modal models — Qwen3.5-Omni (March 2026), Gemini Omni Flash, and others natively unify text, image, audio, and video understanding within a single framework. A latency-optimized architecture that handles only static images has diminishing practical relevance: real-world edge deployments (robotics, AR glasses, in-vehicle systems) increasingly demand video-stream and speech understanding. The paper itself acknowledges this limitation as "future work," but the problem is deeper — when omni-modality is already the industry baseline, the contribution is partially obsolete at the time of submission, not a direction for future exploration.

  2. The paper acknowledges that RL post-training remains "unexplored" but treats this as an optional follow-up. By 2026, GRPO/RLVR-based post-training has become the standard final step for MLLMs to reach state-of-the-art performance (e.g., MixGRPO at ECCV 2026). The paper claims DeCoVE-3B matches Qwen3.5-VL 2B and establishes a new Pareto frontier, but this conclusion is drawn from SFT-stage comparisons only. The critical question is left open: after RL post-training, does the performance gap between encoder-free and modular architectures narrow or widen? If modular MLLMs benefit disproportionately from RL — which is plausible given their access to high-quality visual representations from pretrained encoders — then DeCoVE's claimed Pareto frontier may dissolve at the true state-of-the-art level. For a paper whose core contribution is establishing a new performance–latency Pareto frontier, omitting this experiment is a foundational gap in the evidence.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer KN4s

We thank the reviewer for their insightful feedback, and we are glad that the reviewer found our work "methodologically rigorous and rare", "architecturally clean", and "adaptable to diverse deployment constraints". Below, we address the questions and concerns of the reviewer.

W1. Image-only scope versus omni-modal models for edge deployment

We agree that real-time interaction latency is important for edge deployments. Since the focus of the paper is on vision-language modeling, we demonstrate the advantages of DeCoVE on real-time interaction by fine-tuning on video instruction tuning data. Because Qwen3.5 Omni is not open sourced, we compared it against Qwen3.5 2B for video understanding. As shown in the table below, DeCoVE can be easily adopted for video understanding and achieves competitive results while significantly reducing prefill latency. We have added the training details in Appendix C.6.

BenchmarkDeCoVE acc.Qwen3.5-2B acc.DeCoVE TTFT (ms)Qwen3.5-2B TTFT (ms)Speedup
ActivityNet-QA66.4754.70173.4220.61.27$\times$
MMVU39.8046.70122.5159.81.30$\times$
MotionBench60.3251.93101.8164.61.62$\times$
MVBench56.1557.3392.6139.71.51$\times$
Average55.6952.67122.6171.21.40$\times$

W2. No RL post-training; does the Pareto frontier hold after RL?

We thank the reviewer for raising this point. We want to first clarify that the Qwen3.5 2B model is the final RL model, not after SFT stage. So the comparison is between SFT DeCoVE and RL Qwen model, where DeCoVE already shows comparable performance.

We agree that RL post-training is now a standard final stage, so we applied it to DeCoVE-3B. Specifically, we train DeCoVE with GSPO [1] for 250 steps on a mix of general vision-language RL data from publicly available sources. More training details can be found in the paper's Appendix C.7. As shown in the table, RL training further improves DeCoVE's performance, showing that DeCoVE is compatible with existing post-training methods.

BenchmarkQwen3.5-2BDeCoVEDeCoVE + GRPO
ScreenSpot-v266.569.972.80 (+2.90)
MMMU-Pro24.926.426.99 (+0.59)
ChartQA78.478.177.80 (-0.30)
CountBenchQA83.884.285.63 (+1.43)
BLINK48.654.955.92 (+1.02)
RealWorldQA65.167.269.28 (+2.08)

[1] Zheng, Chujie et al. "Group Sequence Policy Optimization." arXiv preprint arXiv:2507.18071 (2025).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of reviewers feedback and authors response

Dear AC and Reviewers,

We sincerely thank you for your constructive feedback and thoughtful engagement with our work. Your comments have substantially improved the paper.

Strengths: Reviewers recognized the simplicity and effectiveness of the proposed DeCoVE architecture and the comprehensive ablations and analysis. Reviewers also highlighted the strong empirical results across a diverse set of tasks, together with low latency.

Summary of revisions (changes are highlighted in blue in the revised PDF):

  • Reviewer KN4s: Added results on RL and video understanding, showing RL post training is effective for DeCoVE and DeCoVE can be easily adopted for video understanding.
  • Reviewer 5n93: Added text-only performance results and end-to-end efficiency measurements.
  • Reviewer 5Xcg: Fixed the reported typos, and added details on the training setup and latency profiling protocol.

As the discussion period ends on 11/18/2026, we would be grateful if reviewers could let us know whether our responses address their concerns. If they do, we kindly ask that you consider updating your assessment. If any concerns remain, we are happy to provide further clarification or results before the deadline.

Thank you again for your time and effort.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.