- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces BLoCE (Block-Causal Encoding), a method for converting pretrained ViT vision encoders for efficient on-device inference. It employs block-causal attention, allowing bidirectional interactions within blocks and causal access to preceding blocks, together with a bounded historical KV cache to enable blockwise execution across encoder layers. A progressive fine-tuning strategy adapts pretrained encoders to this attention pattern while preserving their architecture and output tokens. Using AIMv2-300M, BLoCE achieves comparable performance across 12 MLLM benchmarks while reporting a 3.2× reduction in vision-encoder latency at 1344 × 1344 resolution on a mobile NPU.
Critical strengths
The paper presents an approach to adapting existing large pretrained vision encoders for efficient inference without requiring expensive pretraining from scratch or extensive architectural modifications. The converted encoder largely preserves its original performance while demonstrating meaningful inference efficiency improvements on mobile hardware.
Critical weaknesses
- Generalization to models beyond AIMv2 is questionable. AIMv2 is pretrained with prefix attention, making it compatible with causal restrictions. It remains unclear whether BLoCE can achieve similar results on fully bidirectional vision encoders such as SigLIP2. Tables 6 and 7 show degradation when applying BLoCE to DeiT and InternViT which were pretrained with fully bidirectional attention. In this setup, even with fine-tuning the performance is not fully recovered unlike MLLM results with AIMv2.
- Missing comparisons with relevant efficient attention mechanisms. There are plenty efficient attention mechanisms that have explored similar blockwise processing. At the very minimum, the proposed method should be compared against established approach for adapting pretrained ViTs to windowed attention, such as ViTDet [1].
[1] - Exploring Plain Vision Transformer Backbones for Object Detection
Questions for authors
- Given AIMv2's prefix-attention pretraining, can the authors demonstrate comparable conversion performance on a fully bidirectional pretrained encoder, such as SigLIP2 or QwenViT? This would significantly strengthen the generality of the proposed approach.
- How does BLoCE compare against established window-attention approaches (such as ViTDet) when applied to the same pretrained encoder with comparable fine-tuning budgets? Such a comparison would help establish the benefits of block-causal attention over existing alternatives.
- Since BLoCE retains all visual tokens, what is the actual improvement in end-to-end MLLM latency, including vision encoding and LLM prefill? This would help establish the practical significance of the reported encoder-level speedup.
6 comments
Response to Reviewer 6ARt (Part 2 / 2): ViTDet-Style Windowed Baseline Comparison and End-to-End MLLM TTFT Speedup (Q2–Q3)
2. Comparison with a ViTDet-Style Window-Attention Configuration (Q2 & Weakness 2)
Direct Answer: Adapting the same AIMv2-300M backbone under the same 5.7M conversion budget at on the Dimensity 9500 NPU (Table 15 protocol), BLoCE obtains 71.0 in 699 ms (0.4 points below Source AIMv2's 71.4 and 0.2 points above the same-data FA-tuned control's 70.8). The ViTDet-style interleaved window/global configuration obtains 70.4 (0.6 points below BLoCE) in 1208 ms, so BLoCE reduces encoder latency by 42.1%. Pure local windows obtain 66.9 in 641 ms; BLoCE gains 4.1 points for 58 ms (9.0%) additional latency.
Evidence and Analysis:
For an apples-to-apples comparison on the same 24-layer AIMv2-300M encoder (W8 dynamic quantization on the Dimensity 9500 via the MediaTek NeuroPilot SDK, Appendix C), we implemented a ViTDet-style configuration with local window attention ( px, 256 tokens/window) in 20 layers and full global attention in layers 6, 12, 18, and 24 (1-indexed; global interaction every six layers). We also evaluated pure local windows across all 24 layers and four tiled baselines ( grids of crops and grids of crops, with and without a crop-resolution global thumbnail; token counts are encoder outputs before the projector):
| Attention Configuration on AIMv2-300M (, Table 15 Protocol) | ViT Output Tokens | 12-Benchmark Mean | NPU Encoder Latency |
|---|---|---|---|
| Full-attention AIMv2 | 9,216 (1.00×) | 71.4 | 2220 ms |
| Tiles (, four crops) | 9,216 (1.00×) | 69.3 | 1381 ms |
| Tiles (, nine crops) | 9,216 (1.00×) | 68.6 | 1103 ms |
| Tiles () + global thumbnail | 11,520 (1.25×) | 69.4 | 1732 ms |
| Tiles () + global thumbnail | 10,240 (1.11×) | 68.8 | 1224 ms |
| Pure local windows (, all 24 layers) | 9,216 (1.00×) | 66.9 | 641 ms |
| ViTDet-style config on AIMv2 (20 local + 4 global layers) | 9,216 (1.00×) | 70.4 | 1208 ms |
| BLoCE () | 9,216 (1.00×) | 71.0 | 699 ms |
- Latency and Quality vs. Windowed Baselines: Pure local windows lack cross-window context (66.9). While the ViTDet-style configuration recovers quality to 70.4, its four global layers (layers 6, 12, 18, and 24) introduce four full-image synchronization barriers: four times during encoding, the NPU must finish all windows across the entire image, materialize full-image activations, and execute global attention across all tokens. By eliminating global barriers, BLoCE allows each block to traverse all 24 layers continuously in 699 ms (−42.1%) while scoring 0.6 points higher.
- Tiled Encoding: All four tiled variants have lower means (68.6–69.4) and higher encoder latency (1103–1732 ms) than BLoCE. The grid is faster but scores lower than , and adding a global thumbnail improves the mean by only 0.1–0.2 points while increasing both encoder latency and output token counts.
3. Improvement in End-to-End MLLM Latency Including LLM Prefill (Q3)
Direct Answer: At , BLoCE reduces end-to-end MLLM Time-to-First-Token (TTFT)—including vision encoding, the MLP projector, and 4B LLM prefill—from 4231 ms to 2739 ms, delivering a 1.54× end-to-end speedup (−1492 ms).
Evidence and Analysis:
| MLLM Pipeline Stage () | Full-Attention MLLM | BLoCE MLLM | Latency Change / Speedup |
|---|---|---|---|
| Vision Encoder Latency | 2220 ms | 699 ms | −1521 ms (3.18× speedup) |
| Projector + LLM Prefill Latency | 2011 ms | 2040 ms | Approximately the same (+29 ms, +1.4%) |
| End-to-End MLLM TTFT | 4231 ms | 2739 ms | −1492 ms (1.54× speedup) |
- Contribution of Encoder Acceleration to TTFT: Measured on the Dimensity 9500 via the MediaTek NeuroPilot SDK (
W8dynamically quantized vision encoders, Appendix C, and on-deviceW4A16quantized Qwen3-4B prefill under identical settings for both pipelines), the vision encoder processes all unmerged patch tokens and accounts for 52.5% of baseline TTFT. Because the projector merges spatial tokens before LLM prefill, the combined projector and prefill latency remains nearly unchanged, translating the 1521 ms encoder saving into a 1492 ms (35.3%) reduction in end-to-end TTFT.
Follow-Up on Our Response
Dear Reviewer 6ARt:
Thank you again for your constructive feedback. We have provided targeted experiments, analyses, and detailed responses addressing each of your concerns, including conversion of the bidirectionally pretrained SigLIP2 encoder, comparisons with ViTDet-style window attention under the same conversion budget, and end-to-end TTFT measurements. Could you please let us know whether these results and explanations resolve all of your concerns, or whether any points require further clarification? We would be happy to address any remaining questions.
Response to Reviewer 6ARt (Part 1 / 2): Conversion of a Bidirectionally Pretrained SigLIP2 Encoder and Vision-Task Transfer (Q1)
We thank Reviewer
6ARtfor the constructive feedback. Below we address each question by presenting our new conversion results on a bidirectionally pretrained vision encoder, our comparison with a ViTDet-style window-attention configuration, and our end-to-end MLLM latency measurements.1. Generalization to a Bidirectionally Pretrained Vision Encoder (Q1 & Weakness 1)
Direct Answer: Yes. We applied Progressive Conversion to the bidirectionally pretrained SigLIP2-derived 300M vision encoder initialized from Qwen3-VL-4B-Instruct. Direct block-causal masking (
BC-direct) lowers the 12-benchmark mean from 71.3 to 66.9. Progressive Conversion recovers it to 71.5 with full history (BC-conv) and 71.1 with bounded history (BLoCE). The bounded-history model is 0.2 points below the full-attention reference and 0.5 below same-budgetFA-tuned(71.6), showing substantial recovery on a bidirectionally pretrained encoder.Evidence and Analysis: We initialized from the official Qwen3-VL-4B-Instruct checkpoint and retained its native 16×16 patch embedding and positional encoding (16-pixel patches vs. 14-pixel for AIMv2), while keeping the merger/projector and LLM frozen. All five configurations use the exact same visual-to-language interface (passing final-layer encoder tokens through the projector to the LLM). Conversion uses the same 5.7M dataset, training budget, input resolution, and token-block partitioning rule as our AIMv2 experiments, with BLoCE inference parameters B=128,C=256,W=128. Full attention is the source reference; FA-tuned continues full-attention training under the same budget. BC-direct applies block-causal masking without conversion training, while BC-conv applies Progressive Conversion; both retain full history at inference. Setting definitions follow Table 14:
BC-direct, Progressive Conversion recovers 4.6 points (BC-conv) and 4.2 points (BLoCE). BLoCE improves overBC-directon all 12 benchmarks by 3.1–5.1 points. Per-benchmark differences are within −0.7 to +0.9 points vs. the full-attention reference and −1.2 to +0.4 points vs. same-budgetFA-tuned. These results confirm effective recovery across individual tasks beyond AIMv2's prefix-pretrained backbone, while full-attention tuning retains a slightly higher overall mean.DeiT-BaseandInternViT-6B): Now Tables 17 and 18 in Appendix E.1 of the revised manuscript, these experiments evaluate transfer beyond our main MLLM setting and identify capability boundaries of block-causal attention, while the SigLIP2 experiment above directly tests a bidirectionally pretrained encoder within an MLLM:76.8%(BC-direct) to82.9%, matching the full-attention baseline. On COCO, the remaining 1.1 box-AP and 1.3 mask-AP gaps (50.5vs.51.6;44.6vs.45.9) show that conversion does not fully preserve dense-prediction performance, which we note as a limitation (Section 5).BC-LPtrains only the classification head, rather than applying Progressive Conversion to the encoder (Appendices E.1 and E.4).