- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies encoder-free multimodal language models under latency-constrained settings. The authors argue that small encoder-free backbones suffer from insufficient visual processing depth and propose DeCoVE, a deep and narrow visual expert that operates in parallel with the text backbone. DeCoVE supports early cross-modal fusion and initialization from pretrained vision-transformer weights. The resulting DeCoVE-3B model is evaluated on 13 visual understanding, document understanding, and grounding benchmarks. It outperforms existing encoder-free baselines and reports substantially lower time-to-first-token than several modular MLLMs. Ablations examine visual depth, layer placement, early fusion, and pretrained initialization.
Critical strengths
-
Practical and well-motivated architecture. DeCoVE addresses an important limitation of small encoder-free MLLMs by increasing visual depth while retaining parallel image–text processing.
-
Strong empirical evaluation. The paper evaluates 13 benchmarks and provides useful ablations on visual depth, layer placement, early fusion, and pretrained initialization. The reported accuracy–latency trade-off is compelling.
Critical weaknesses
-
Some central implementation details appear inconsistent. Section 4.1 specifies sequential and convolutions, implying an effective stride of 512, which seems incompatible with the reported visual-token counts. Section 5.1 also describes as a higher learning rate than . These specifications need correction before the model can be reproduced.
-
The latency and training-data protocols are insufficiently documented. Since efficiency is a central claim, the paper should report batch size, precision, warm-up runs, repetitions, variance, actual token lengths, and the components included in TTFT. The pretraining and instruction-tuning datasets are described only by broad categories, making contamination and controlled comparison difficult to assess.
Questions for authors
-
What are the exact kernel sizes and strides of the two image-embedding convolutions? How do they produce up to 2,560 visual tokens?
-
Is the learning rate for the newly initialized layers really , or is this a typographical error?
-
Could the authors provide the complete TTFT protocol, including batch size, precision, warm-up, number of runs, variance, and whether preprocessing is included?
-
What datasets constitute the 3.8M VQA, 1.4M OCR, and 15M instruction-tuning samples? Were evaluation-set overlap and contamination checked?
-
Given that DeCoVE contains a 24-layer vision-specific pathway initialized from Qwen-ViT, how do the authors distinguish it conceptually from a parallel or interleaved modular vision encoder?
Rebuttal to Reviewer 5Xcg
We thank the reviewer for the valuable feedback. We appreciate that the reviewer found our architecture "practical and well-motivated" with "strong empirical evaluation", and that the "accuracy–latency trade-off is compelling". We respond to the issues and questions below.
W1. Inconsistent implementation details on convolution
Thank you for catching this. The text in Section 4.1 was miswritten. Conv1 uses a 16×16 kernel with stride 16, and Conv2 uses a 2×2 kernel with stride 2. The total effective stride is therefore 32, and each visual token covers a 32×32 pixel region. A budget of 2,560 visual tokens corresponds to images of about 2.6 MP; for example, a 1280×2048 image gives a 40×64 grid of 2,560 tokens. For images that exceed this 2,560-token cap, we first resize the image so that it fits within the budget. We have corrected Section 4.1 in the revised PDF.
Q2. Is the 4×10−8 learning rate a typo?
Yes, it is a typo. The correct learning rate is 8×10−4. We have corrected it in the revised PDF.
Q3. Complete TTFT protocol
The TTFT protocol uses a batch size of 1 and BF16 precision. For warm-up, we run 3 untimed generations prior to profiling. Then, for each sample, we run 1 untimed generation before the 3 timed prefills, so that the kernels for the input image's shape are ready. We profile all samples of each evaluation, running the prefill 3 times per sample; we take the median of the 3 runs for each sample and then average over the samples. The standard error of the mean is 0.53 ms for DeCoVE-3B (26.16±0.53 ms), compared with 44.63±1.44 ms for Qwen3.5 2B. Image preprocessing and text tokenization are excluded from the profiling. We have added these details in Appendix D.1.
Q4. Composition of the VQA, OCR and instruction-tuning data; contamination checks
The 3.8M VQA data for pretraining are sampled from LLaVA-665k [1], Molmo2 [2], PixMo-Points [3], and MGrounding-630k [4]. The 1.4M OCR data are mostly from FinePDF document data and in-house rendered text datasets for natural-image OCR capabilities. The 15M instruction tuning data are sampled from various publicly available sources; notable sources include LLaVA-OneVision [5], Cambrian [6], and Molmo2 [2]. For all of our training data, we perform exact deduplication against the evaluation sets to remove contamination. We will release details on the data composition after paper acceptance and internal review.
Q5. Conceptual difference from a parallel or interleaved modular vision encoder
Thank you for raising this question. DeCoVE can indeed be viewed as a form of parallel or interleaved modular vision encoder. Our contribution is not to introduce a fundamentally separate category, but to study and refine this broader design space in the context of visual experts in encoder-free MLLMs. Specifically, we address four design questions:
Thus, DeCoVE presents a specific design for a parallel vision encoder, motivated by the advantages and disadvantages of encoder-free MLLMs.
[1] Liu et al. LLaVA-1.5. CVPR 2024.
[2] Clark et al. Molmo2. CVPR 2026.
[3] Deitke et al. Molmo and PixMo. CVPR 2025.
[4] Li et al. Migician. ACL Findings 2025.
[5] Li et al. LLaVA-OneVision. TMLR 2025.
[6] Tong et al. Cambrian-1. NeurIPS 2024.
[7] Alayrac et al. Flamingo. NeurIPS 2022.