acceptodds
Under review as a conference paper at ICLR 2027

Block-Causal Encoding: Converting Large MLLM Encoders for Efficient On-Device Inference

Abstract

Large pretrained vision encoders provide the detailed representations needed by Multimodal Large Language Models (MLLMs), but high-resolution encoding is costly on mobile Neural Processing Units (NPUs). Beyond its quadratic computation, their full attention couples image regions across layers, making it difficult to complete encoding in small blocks suited to limited on-chip memory. We introduce BLoCE (Block-Causal Encoding), which preserves the original ViT layer structure and standard operators while restructuring attention for blockwise execution. It features block-causal attention allowing each image block to complete all encoder layers without waiting for future blocks and a bounded historical key-value cache supplying context for subsequent blocks. This organizes encoding around smaller intermediate features and attention inputs while retaining every output token. We further design Progressive Conversion to adapt the pretrained encoder to the new attention pattern through post-training, avoiding encoder pretraining from scratch. Based on AIMv2-300M, BLoCE achieves a comparable performance across 12 MLLM benchmarks. On a MediaTek Dimensity 9500 NPU, it reduces encoder latency at from 2220 ms to 699 ms, a speedup. These results demonstrate the value of restructuring attention for efficient high-resolution inference while reusing large pretrained vision encoders.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision transformer, MLLM, mobile-friendly, NPU

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6ARtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces BLoCE (Block-Causal Encoding), a method for converting pretrained ViT vision encoders for efficient on-device inference. It employs block-causal attention, allowing bidirectional interactions within blocks and causal access to preceding blocks, together with a bounded historical KV cache to enable blockwise execution across encoder layers. A progressive fine-tuning strategy adapts pretrained encoders to this attention pattern while preserving their architecture and output tokens. Using AIMv2-300M, BLoCE achieves comparable performance across 12 MLLM benchmarks while reporting a 3.2× reduction in vision-encoder latency at 1344 × 1344 resolution on a mobile NPU.

Critical strengths

The paper presents an approach to adapting existing large pretrained vision encoders for efficient inference without requiring expensive pretraining from scratch or extensive architectural modifications. The converted encoder largely preserves its original performance while demonstrating meaningful inference efficiency improvements on mobile hardware.

Critical weaknesses
  1. Generalization to models beyond AIMv2 is questionable. AIMv2 is pretrained with prefix attention, making it compatible with causal restrictions. It remains unclear whether BLoCE can achieve similar results on fully bidirectional vision encoders such as SigLIP2. Tables 6 and 7 show degradation when applying BLoCE to DeiT and InternViT which were pretrained with fully bidirectional attention. In this setup, even with fine-tuning the performance is not fully recovered unlike MLLM results with AIMv2.
  2. Missing comparisons with relevant efficient attention mechanisms. There are plenty efficient attention mechanisms that have explored similar blockwise processing. At the very minimum, the proposed method should be compared against established approach for adapting pretrained ViTs to windowed attention, such as ViTDet [1].

[1] - Exploring Plain Vision Transformer Backbones for Object Detection

Questions for authors
  1. Given AIMv2's prefix-attention pretraining, can the authors demonstrate comparable conversion performance on a fully bidirectional pretrained encoder, such as SigLIP2 or QwenViT? This would significantly strengthen the generality of the proposed approach.
  2. How does BLoCE compare against established window-attention approaches (such as ViTDet) when applied to the same pretrained encoder with comparable fine-tuning budgets? Such a comparison would help establish the benefits of block-causal attention over existing alternatives.
  3. Since BLoCE retains all visual tokens, what is the actual improvement in end-to-end MLLM latency, including vision encoding and LLM prefill? This would help establish the practical significance of the reported encoder-level speedup.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 6ARt (Part 1 / 2): Conversion of a Bidirectionally Pretrained SigLIP2 Encoder and Vision-Task Transfer (Q1)

We thank Reviewer 6ARt for the constructive feedback. Below we address each question by presenting our new conversion results on a bidirectionally pretrained vision encoder, our comparison with a ViTDet-style window-attention configuration, and our end-to-end MLLM latency measurements.

1. Generalization to a Bidirectionally Pretrained Vision Encoder (Q1 & Weakness 1)

Direct Answer: Yes. We applied Progressive Conversion to the bidirectionally pretrained SigLIP2-derived 300M vision encoder initialized from Qwen3-VL-4B-Instruct. Direct block-causal masking (BC-direct) lowers the 12-benchmark mean from 71.3 to 66.9. Progressive Conversion recovers it to 71.5 with full history (BC-conv) and 71.1 with bounded history (BLoCE). The bounded-history model is 0.2 points below the full-attention reference and 0.5 below same-budget FA-tuned (71.6), showing substantial recovery on a bidirectionally pretrained encoder.

Evidence and Analysis: We initialized from the official Qwen3-VL-4B-Instruct checkpoint and retained its native 16×1616 \times 16 patch embedding and positional encoding (16-pixel patches vs. 14-pixel for AIMv2), while keeping the merger/projector and LLM frozen. All five configurations use the exact same visual-to-language interface (passing final-layer encoder tokens through the projector to the LLM). Conversion uses the same 5.7M dataset, training budget, input resolution, and token-block partitioning rule as our AIMv2 experiments, with BLoCE inference parameters B=128,C=256,W=128B=128, C=256, W=128. Full attention is the source reference; FA-tuned continues full-attention training under the same budget. BC-direct applies block-causal masking without conversion training, while BC-conv applies Progressive Conversion; both retain full history at inference. Setting definitions follow Table 14:

BenchmarkFull AttentionFA-tunedBC-directBC-convBLoCE
CCBench58.759.454.058.958.6
ChartQA84.584.778.984.083.8
ChineseOCRBench81.782.578.182.382.6
DocVQA91.992.388.592.591.6
HallusionBench59.459.356.160.159.7
InfoVQA74.673.969.174.574.2
MMBench77.677.472.877.977.4
MMMU50.851.047.451.250.7
MMT-Bench62.062.556.961.861.3
OCRBench79.980.975.080.480.1
RealWorldQA68.268.163.967.668.1
SEEDBench2+66.166.862.366.465.6
12-Benchmark Mean71.371.666.971.571.1
  1. Conversion Recovery on a Bidirectional Encoder: Relative to the 4.4-point drop under BC-direct, Progressive Conversion recovers 4.6 points (BC-conv) and 4.2 points (BLoCE). BLoCE improves over BC-direct on all 12 benchmarks by 3.1–5.1 points. Per-benchmark differences are within −0.7 to +0.9 points vs. the full-attention reference and −1.2 to +0.4 points vs. same-budget FA-tuned. These results confirm effective recovery across individual tasks beyond AIMv2's prefix-pretrained backbone, while full-attention tuning retains a slightly higher overall mean.
  2. Interpreting the Submitted Tables 6 and 7 (DeiT-Base and InternViT-6B): Now Tables 17 and 18 in Appendix E.1 of the revised manuscript, these experiments evaluate transfer beyond our main MLLM setting and identify capability boundaries of block-causal attention, while the SigLIP2 experiment above directly tests a bidirectionally pretrained encoder within an MLLM:
    • On ImageNet classification (revised Table 17), adapting bidirectionally pretrained DeiT-Base recovers top-1 accuracy from 76.8% (BC-direct) to 82.9%, matching the full-attention baseline. On COCO, the remaining 1.1 box-AP and 1.3 mask-AP gaps (50.5 vs. 51.6; 44.6 vs. 45.9) show that conversion does not fully preserve dense-prediction performance, which we note as a limitation (Section 5).
    • InternViT-6B (revised Table 18) uses frozen-backbone linear probing: BC-LP trains only the classification head, rather than applying Progressive Conversion to the encoder (Appendices E.1 and E.4).
AuthorsOpenReview ↗

Response to Reviewer 6ARt (Part 2 / 2): ViTDet-Style Windowed Baseline Comparison and End-to-End MLLM TTFT Speedup (Q2–Q3)

2. Comparison with a ViTDet-Style Window-Attention Configuration (Q2 & Weakness 2)

Direct Answer: Adapting the same AIMv2-300M backbone under the same 5.7M conversion budget at 1344×13441344 \times 1344 on the Dimensity 9500 NPU (Table 15 protocol), BLoCE obtains 71.0 in 699 ms (0.4 points below Source AIMv2's 71.4 and 0.2 points above the same-data FA-tuned control's 70.8). The ViTDet-style interleaved window/global configuration obtains 70.4 (0.6 points below BLoCE) in 1208 ms, so BLoCE reduces encoder latency by 42.1%. Pure local windows obtain 66.9 in 641 ms; BLoCE gains 4.1 points for 58 ms (9.0%) additional latency.

Evidence and Analysis: For an apples-to-apples comparison on the same 24-layer AIMv2-300M encoder (W8 dynamic quantization on the Dimensity 9500 via the MediaTek NeuroPilot SDK, Appendix C), we implemented a ViTDet-style configuration with 16×1616 \times 16 local window attention (224×224224 \times 224 px, 256 tokens/window) in 20 layers and full global attention in layers 6, 12, 18, and 24 (1-indexed; global interaction every six layers). We also evaluated pure 16×1616\times16 local windows across all 24 layers and four tiled baselines (2×22\times2 grids of 672×672672\times672 crops and 3×33\times3 grids of 448×448448\times448 crops, with and without a crop-resolution global thumbnail; token counts are encoder outputs before the projector):

Attention Configuration on AIMv2-300M (1344×13441344 \times 1344, Table 15 Protocol)ViT Output Tokens12-Benchmark MeanNPU Encoder Latency
Full-attention AIMv29,216 (1.00×)71.42220 ms
Tiles (2×22\times2, four 672×672672\times672 crops)9,216 (1.00×)69.31381 ms
Tiles (3×33\times3, nine 448×448448\times448 crops)9,216 (1.00×)68.61103 ms
Tiles (2×22\times2) + global thumbnail11,520 (1.25×)69.41732 ms
Tiles (3×33\times3) + global thumbnail10,240 (1.11×)68.81224 ms
Pure local windows (16×1616 \times 16, all 24 layers)9,216 (1.00×)66.9641 ms
ViTDet-style config on AIMv2 (20 local + 4 global layers)9,216 (1.00×)70.41208 ms
BLoCE (B=128,C=256B=128, C=256)9,216 (1.00×)71.0699 ms
  1. Latency and Quality vs. Windowed Baselines: Pure local windows lack cross-window context (66.9). While the ViTDet-style configuration recovers quality to 70.4, its four global layers (layers 6, 12, 18, and 24) introduce four full-image synchronization barriers: four times during encoding, the NPU must finish all windows across the entire image, materialize full-image N×DN \times D activations, and execute global attention across all N=9216N=9216 tokens. By eliminating global barriers, BLoCE allows each block to traverse all 24 layers continuously in 699 ms (−42.1%) while scoring 0.6 points higher.
  2. Tiled Encoding: All four tiled variants have lower means (68.6–69.4) and higher encoder latency (1103–1732 ms) than BLoCE. The 3×33\times3 grid is faster but scores lower than 2×22\times2, and adding a global thumbnail improves the mean by only 0.1–0.2 points while increasing both encoder latency and output token counts.

3. Improvement in End-to-End MLLM Latency Including LLM Prefill (Q3)

Direct Answer: At 1344×13441344 \times 1344, BLoCE reduces end-to-end MLLM Time-to-First-Token (TTFT)—including vision encoding, the MLP projector, and 4B LLM prefill—from 4231 ms to 2739 ms, delivering a 1.54× end-to-end speedup (−1492 ms).

Evidence and Analysis:

MLLM Pipeline Stage (1344×13441344 \times 1344)Full-Attention MLLMBLoCE MLLMLatency Change / Speedup
Vision Encoder Latency2220 ms699 ms−1521 ms (3.18× speedup)
Projector + LLM Prefill Latency2011 ms2040 msApproximately the same (+29 ms, +1.4%)
End-to-End MLLM TTFT4231 ms2739 ms−1492 ms (1.54× speedup)
  1. Contribution of Encoder Acceleration to TTFT: Measured on the Dimensity 9500 via the MediaTek NeuroPilot SDK (W8 dynamically quantized vision encoders, Appendix C, and on-device W4A16 quantized Qwen3-4B prefill under identical settings for both pipelines), the vision encoder processes all N=9216N=9216 unmerged patch tokens and accounts for 52.5% of baseline TTFT. Because the projector merges spatial tokens before LLM prefill, the combined projector and prefill latency remains nearly unchanged, translating the 1521 ms encoder saving into a 1492 ms (35.3%) reduction in end-to-end TTFT.
AuthorsOpenReview ↗

Follow-Up on Our Response

Dear Reviewer 6ARt:

Thank you again for your constructive feedback. We have provided targeted experiments, analyses, and detailed responses addressing each of your concerns, including conversion of the bidirectionally pretrained SigLIP2 encoder, comparisons with ViTDet-style window attention under the same conversion budget, and end-to-end TTFT measurements. Could you please let us know whether these results and explanations resolve all of your concerns, or whether any points require further clarification? We would be happy to address any remaining questions.

Reviewer_AubGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BLoCE, a way to convert a pretrained full-attention ViT encoder for blockwise execution on mobile NPUs without changing its layers, operators, or output tokens. It has three parts. Block-causal attention lets tokens attend bidirectionally within a block and to earlier blocks only, so each block can pass through all layers before the next one starts. A bounded historical KV cache makes attention cost linear in the number of tokens. Progressive Conversion is a post-training recipe that tunes only the attention projections, shrinks the block size gradually during training, and samples block sizes at random. Applied to AIMv2-300M in AIMv2-Qwen3-4B, BLoCE scores a 12-benchmark mean of 74.0, against 73.9 for the source model. On a Dimensity 9500 NPU it cuts encoder latency at 1344 ×\times 1344 from 2220 ms to 699 ms.

Critical strengths
  1. A practical problem, a deployment-friendly design, and real-device evidence. High-resolution encoding is a real bottleneck for on-device MLLMs. BLoCE uses only standard operators and keeps the original parameter shapes, so it fits a vendor toolchain and its fused attention operator without custom kernels. Designs that need sparse or linear-attention kernels often cannot claim this. The speedup is measured on a commercial mobile NPU rather than estimated from FLOPs.

  2. Cheap conversion and flexible deployment. Conversion needs only a few million samples and about 1.9k GPU-hours, orders of magnitude less than pretraining the encoder from scratch, and a fraction of the data already recovers most of the lost accuracy. Thanks to block-size sampling, a single converted checkpoint runs at several block sizes, including full attention for small images, with almost no change in quality. Deployment can therefore pick the fastest mode per resolution without retraining.

Critical weaknesses
  1. Why block causality, rather than the simpler solutions already in use? Tiling and local window attention already give linear cost with independent working sets, yet the paper compares only against whole-image full attention. Its own results suggest BLoCE behaves like local attention: inference-time eviction costs almost nothing, and the history mechanism is never ablated. If that is the case, the novelty over known components largely disappears.

  2. Causal ordering is a questionable prior for images, and the evaluation does not test what it might break. Under a raster order, early regions never see the rest of the image, which conflicts with the paper's own motivation of high-resolution, global understanding. The "comparable quality" evidence comes from a noisy benchmark mean at native resolutions, where the restriction rarely binds. The gaps that remain on detection and segmentation point the other way.

  3. The speedup is not separated from one specific toolchain. All evidence comes from one NPU and one SDK, and the full-attention baseline times out where quadratic scaling predicts only a few seconds. With no memory, energy or end-to-end measurements, it is unclear whether the gain is a general property of the method or an artifact of a single chip's compiler.

Questions for authors
  1. Under the same conversion budget and on the same NPU, how does BLoCE compare with tiled encoding and with a window-attention conversion of the same encoder, at matched accuracy or matched latency?
  2. What does the historical KV cache actually contribute? Concretely, how much does quality drop if only the previous block is kept, or if the importance–redundancy selection is replaced by simple recency?
  3. Does quality hold when the image is evaluated at a fixed high resolution, where the restriction truly binds? Does accuracy depend on whether the content a question asks about appears early or late in the scan order?
  4. Where do the savings come from in practice? A breakdown of encoder latency, peak memory, and end-to-end TTFT would show this. How is the attention-based history selection executed on an NPU whose fused attention operator does not expose attention maps?
6 comments
AuthorsOpenReview ↗

Response to Reviewer AubG (Part 1 / 2): Comparisons with Tiled/Windowed Baselines and Historical KV Cache Ablations (Q1–Q2)

We thank Reviewer AubG for the rigorous questions and constructive feedback. Below we address each question directly with our new experimental and device results.

1. Comparison with Tiled Encoding and Window-Attention Conversions (Q1 & Weakness 1)

Direct Answer: Under the same 5.7M conversion budget at 1344×13441344 \times 1344 on the Dimensity 9500 NPU (Table 15 protocol), BLoCE (71.0 mean, 699 ms) is 0.4 points below Source AIMv2 (71.4) and 0.2 points above the same-data FA-tuned control (70.8). It improves the 12-benchmark mean by 1.6–2.4 points over all four tiled baselines at 36.6–59.6% lower latency, gains 4.1 points over pure 16×1616\times16 local windows for 58 ms (9.0%) extra latency, and outperforms the ViTDet-style configuration by 0.6 points at 42.1% lower latency.

Evidence and Analysis: All baselines adapt the same AIMv2-300M backbone (W8 dynamic quantization via the MediaTek NeuroPilot SDK, Table 15). Tiled variants optionally add a resized full-image thumbnail matching a single crop's resolution (6722672^2 or 4482448^2), and the ViTDet-style setup uses 16×1616\times16 (2242224^2 px, 256-token) non-overlapping local windows in 20 layers and full global attention in layers 6, 12, 18, and 24 (token counts denote encoder outputs before the projector):

Encoder Configuration (1344×13441344 \times 1344, Table 15 Protocol)Output Tokens12-Benchmark MeanNPU Encoder Latency
Full-attention AIMv29,216 (1.00×)71.42220 ms
Tiles (2×22\times2, four 672×672672\times672 crops)9,216 (1.00×)69.31381 ms
Tiles (3×33\times3, nine 448×448448\times448 crops)9,216 (1.00×)68.61103 ms
Tiles (2×22\times2) + global thumbnail11,520 (1.25×)69.41732 ms
Tiles (3×33\times3) + global thumbnail10,240 (1.11×)68.81224 ms
Pure local windows (16×1616 \times 16, all 24 layers)9,216 (1.00×)66.9641 ms
ViTDet-style config on AIMv2 (20 local + 4 global layers)9,216 (1.00×)70.41208 ms
BLoCE (B=128,C=256B=128, C=256)9,216 (1.00×)71.0699 ms
  • Local and ViTDet-style windows: Pure local windows lack cross-window context (66.9). In the ViTDet-style setup, the four global layers recover the mean to 70.4, but introduce four full-image synchronization barriers: before each global layer can run, all windows must complete the preceding local layers and supply full-image activations. BLoCE lets each block complete all 24 layers continuously, cutting latency by 42.1%.
  • Tile size and global thumbnails: Moving from 2×22\times2 to 3×33\times3 tiles reduces latency by 20.1% at a 0.7-point accuracy cost, while adding a global thumbnail improves the mean by only 0.1–0.2 points at substantial latency (+351 ms / +121 ms) and token (+25% / +11.1%) overhead.

2. Contribution of the Historical KV Cache and Selection Rule (Q2 & Weakness 1)

Direct Answer: Removing the historical KV cache (C = 0) drops the 12-benchmark mean by 6.4 points (74.0 →\to 67.6), keeping only the previous block (C = 128) drops it by 3.9 points (70.1), and replacing our importance–redundancy selection with simple recency at the same budget (C = 256) drops it by 2.6 points (71.4).

Evidence and Analysis: We evaluated the same converted checkpoint at B=128B=128, changing only the inference-time history configuration (distinct from the separately converted 256-token local-window baseline in Q1):

Historical Context Configuration (B=128B=128)Retained History per Layer/Head12-Benchmark MeanΔ\Delta vs. No History
1. No historyC=0C = 0 (current 128 tokens)67.6—
2. Previous block onlyC=128C = 128 (W=128W=128 recent tokens)70.1+2.5
3. Simple recencyC=256C = 256 (256 recent tokens)71.4+3.8
4. BLoCE selection (Importance–Redundancy)C=256C = 256 (W=128W=128 recent + 128 selected)74.0+6.4
5. Full history (BC-conv)Unbounded (C=NC = N)74.1+6.5
  1. BLoCE does not reduce to local attention: Because C = 0 causes a 6.4-point drop, the 0.1-point gap between unbounded history (74.1) and BLoCE (74.0) reflects the effectiveness of the C=256C=256 selected cache rather than insensitivity to historical context.
  2. Importance–redundancy selection vs. recency: Keeping only the most recent one or two blocks discards explicit KV entries from earlier regions, even though recent representations carry some indirect context. Selecting C−W=128C-W=128 entries from up to M≤C+B−W=256M \le C+B-W = 256 older candidates via importance–redundancy scoring recovers this long-range context (+2.6 points over simple recency).
AuthorsOpenReview ↗

Response to Reviewer AubG (Part 2 / 2): Fixed High-Resolution Quality, Scan-Order Sensitivity, and On-Device Breakdown (Q3–Q4)

3. Quality at Fixed High Resolution and Sensitivity to Scan Order (Q3 & Weakness 2)

Direct Answer: Under the fixed 1344×13441344 \times 1344 quantized evaluation (Table 15 protocol, N=9216N=9216, 72 blocks), BLoCE (71.0) remains within 0.4 points of Source AIMv2 (71.4) and 0.2 points above the same-data FA-tuned control (70.8). In a floating-point DocVQA analysis at 1344×13441344 \times 1344 grouped by answer position, original-order BLoCE stays within 0.4 points of full attention across early, middle, and late groups (−0.1 / +0.4 / −0.2 points), while reversing the block order changes the scores by −0.2 / −0.7 / −0.5 points (scoring lower in all three groups).

Evidence and Analysis:

  1. Per-Benchmark Results at Fixed 1344×13441344 \times 1344 (Table 15 W8 quantized protocol; abbreviations as in Table 1):
SettingCCCQAC-OCRDVQAHallu.IVQAMMBMMMUMMTOCRBR-QASEED2+MeanLatency
AIMv259.383.982.392.559.574.278.250.562.479.567.866.271.42220 ms
FA-tuned58.583.182.491.558.374.377.649.461.979.567.566.070.82220 ms
BLoCE58.283.482.791.959.274.577.250.661.179.268.465.571.0699 ms

Across all 12 tasks, differences remain within [−1.3,+0.6][-1.3, +0.6] vs. AIMv2 and [−0.8,+1.2][-0.8, +1.2] vs. FA-tuned (under AndesVL resolutions, reversed and serpentine block orders also remain within 0.3–0.4 mean points at 73.6 and 73.7 vs. 74.0, with changes in [−1.1,+0.4][-1.1, +0.4] on the five reported tasks; see wFyu Q2).

  1. Answer-Position Evaluation on DocVQA (floating point, 1344×13441344 \times 1344): Grouping questions by the block containing the answer-region center in the original scan—Early (blocks 1–24), Middle (25–48), or Late (49–72), kept fixed across runs—reversing the block sequence moves originally early answer regions to late scan positions and vice versa (preserving positional embeddings and restoring output order before the projector):
Answer Position in Original Block OrderFull AttentionBLoCE, Original OrderBLoCE, Reversed OrderReversed − Original
Early94.194.093.8−0.2
Middle93.694.093.3−0.7
Late93.893.693.1−0.5
  1. Detection and Segmentation (submitted Table 6; revised Table 17): The remaining 1.1 box-AP and 1.3 mask-AP gaps (50.5 vs. 51.6; 44.6 vs. 45.9) reflect capability boundaries of block-causal encoding on dense prediction (Section 5, Limitations); for generality across MLLM backbones, see our SigLIP2 experiment in 6ARt Q1.

4. Practical Savings Breakdown and NPU Execution of Historical KV Selection (Q4 & Weakness 3)

Direct Answer: At 1344×13441344 \times 1344 on the Dimensity 9500, BLoCE reduces end-to-end MLLM TTFT from 4231 ms (2220 ms W8 encoder + 2011 ms projector & W4A16 Qwen3-4B prefill) to 2739 ms (699 ms + 2040 ms, 1.54× speedup), cuts vision-encoder device peak memory from 3.50 GiB (3584 MiB) to 1.08 GiB (1106 MiB, −69.1%) (MediaTek NeuroPilot Runtime), and lowers external DRAM traffic by 54%. Within BLoCE's 699 ms encoder latency, the 24-layer block forward passes take 648 ms, and historical KV scoring, selection, data movement, and cache updates take 51 ms (0.71 ms/block).

Evidence and Implementation Details:

  1. Second Platform (Dimensity 9400): Using the NeuroPilot SDK on a MediaTek Dimensity 9400 NPU, encoder latency at 1344×13441344 \times 1344 drops from 2413 ms to 814 ms (2.96×), reproducing the speedup across two MediaTek NPU generations; as cross-vendor validation requires dedicated hardware and vendor support, we leave cross-vendor generalization to future work.
  2. 1792×17921792 \times 1792 Timeout & Energy Scope: At 1792×17921792 \times 1792 (N=16,384N=16,384), BLoCE completes in 778 ms whereas full attention times out at 10,000 ms. While we lack hardware/compiler visibility to confirm the cause (memory footprint is a plausible factor), our speedup claim does not depend on this timeout and is supported by the completed 4482–13442448^2\text{--}1344^2 sweep in Table 2 (crossing over at 8962896^2 and reaching 3.2× at 134421344^2). As on-device energy profiling requires tooling not yet established, we leave energy evaluation to future work (wFyu Q1).
  3. Historical KV Selection Alongside the Fused Attention Operator: The fused operator computes attention outputs, while history importance is obtained via a separate QK–softmax reduction on the same query/key inputs, so the fused operator need not expose attention maps.

Thank you for the author's response, it has resolved most of my concerns. My remaining questions mainly focus on Q4/Weakness 3. The newly added time-consuming splitting and the results on Dimensity 9400 are also commendable, but both devices still belong to the MediaTek NeuroPilot toolchain. If the paper advocates a universal visual encoder acceleration methodology, I would like to see evidence or analysis that it can be transferred to hardware with different rooflines and memory hierarchies, rather than being limited to the same vendor system. If the actual positioning is deployment optimization research for MediaTek, please express this proposition more clearly. As of the current response, Q4 has only received partial answers.

Reviewer_wFyuOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces BLoCE, a framework for efficient high-resolution vision encoding on mobile devices. It replaces full attention with block-causal attention and a bounded KV cache. This enables blockwise execution while preserving the original vit architecture. A progressive conversion strategy adapts pretrained encoders without training from scratch. Experiments demonstrate comparable performance across 12 MLLM benchmarks and a 3.2x encoder speedup on a mobile NPU.

Critical strengths
  1. Effective architectural adaptation with minimal performance degradation. BLoCE provides a practical approach to converting pretrained vision encoders into blockwise architectures without modifying their fundamental structure. The progressive conversion strategy successfully preserves downstream performance across 12 MLLM benchmarks, demonstrating that substantial changes to the attention mechanism can be introduced without compromising representation quality.

  2. Convincing efficiency improvements on real hardware. Unlike approaches relying primarily on theoretical complexity analysis, the paper demonstrates substantial latency improvements on an actual mobile NPU, achieving a 3.2x speedup at high resolution. The experiments also identify the resolution-dependent trade-offs of blockwise execution, strengthening the practical relevance of the contribution.

Critical weaknesses
  1. Limited hardware generalizability and incomplete efficiency analysis. The latency evaluation is restricted to a single mobile NPU platform, making it difficult to assess whether the reported improvements generalize to other hardware architectures. Furthermore, despite the emphasis on memory-efficient execution, the paper does not provide direct measurements of peak memory usage, memory traffic, or energy consumption. These measurements would strengthen the claimed practical benefits.

  2. Limited analysis of spatial dependencies and conversion costs. Block-causal attention introduces an asymmetric dependency between image regions, since earlier blocks cannot attend to later ones. The paper does not thoroughly investigate how spatial ordering affects representation quality, particularly for tasks requiring global spatial context. Additionally, the conversion process requires substantial computational resources (approximately 60 hours on 32 GPUs), which may limit the practical accessibility of the approach.

Questions for authors
  1. Can the authors provide direct measurements of peak memory usage, memory traffic, and energy consumption? Additionally, how dependent are the reported latency improvements on the specific NPU architecture and optimized attention operators? These results would help substantiate the practical efficiency claims.

  2. Since block-causal attention introduces asymmetric dependencies between image regions, how sensitive is the model to the ordering of spatial blocks? Could the authors evaluate alternative traversal orders and their impact on downstream performance, particularly for tasks requiring global spatial reasoning?

  3. The progressive conversion requires approximately 60 hours on 32 GPUs. How does performance vary with smaller conversion datasets or shorter training schedules?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wFyu (Part 1 / 2): Device Peak Memory, Traffic, Energy Scope, and Second NPU Evaluation (Q1)

We thank Reviewer wFyu for the positive evaluation of our method's soundness, clarity, and hardware efficiency. Below we address each question by first stating our direct findings and then providing the supporting measurements and analysis.

1. Peak Memory, Memory Traffic, Energy Evaluation, and Second NPU Results (Q1)

Direct Answer: On the Dimensity 9500 NPU at 1344×13441344 \times 1344 (vision encoder only), BLoCE reduces device peak memory from 3.50 GiB to 1.08 GiB (−69.1%) and external-memory traffic by 54%. In addition, we reproduce the latency improvement on a second mobile NPU (MediaTek Dimensity 9400), achieving a 2.96× speedup (2413 ms →\to 814 ms).

Evidence and Analysis:

Hardware Metric (1344×13441344 \times 1344, N=9216N=9216, Vision Encoder Only)Full Attention (B=NB=N)BLoCE (B=128,C=256B=128, C=256)Improvement
Dimensity 9500 Encoder Latency2220 ms699 ms3.18× speedup (−68.5%)
Dimensity 9500 Device Peak Memory3.50 GiB1.08 GiB69.1% reduction
Dimensity 9500 External-Memory Traffic100% (Ref.)46%54% reduction
Dimensity 9400 Encoder Latency (2nd NPU)2413 ms814 ms2.96× speedup (−66.3%)
  1. Device Peak Memory and External Traffic Scope: All measurements above cover vision encoder execution on the mobile device (W8 dynamic quantization via the MediaTek NeuroPilot SDK, following Appendix C), excluding the LLM. Peak device memory is obtained through the MediaTek NeuroPilot Runtime and decreases from 3584 MiB (3.50 GiB) to 1106 MiB (1.08 GiB), saving 2478 MiB. Tables 12–13 describe logical tensor sizes, whereas the peak-memory and traffic values above are direct device measurements. Actual buffer placement and reuse depend on the vendor compiler and runtime (Appendix B.4).
  2. Second Platform and Scope of Hardware Generalization: Both the Dimensity 9500 and 9400 use the MediaTek NeuroPilot SDK, so the 1344×13441344 \times 1344 latency reduction on the Dimensity 9400 (2.96×) reproduces the speedup across two MediaTek NPU generations. As cross-vendor deployment and validation require substantial engineering resources, dedicated hardware, and vendor-specific technical support, our hardware evaluation is limited to two MediaTek NPU generations, and cross-vendor generalization (e.g., Qualcomm deployment) is left to future work.
  3. Energy Evaluation: Our direct measurements demonstrate improvements in encoder latency, peak memory, memory traffic, and end-to-end TTFT. These results do not imply energy savings, though we agree that energy per inference would provide a useful complementary measure of deployment efficiency. Reliable on-device energy measurements require additional profiling software and technical support. As of the submission of this response, we have not yet established and validated the required measurement setup and may therefore leave this evaluation to future work.
AuthorsOpenReview ↗

Response to Reviewer wFyu (Part 2 / 2): Spatial Block Traversal Orders, Answer-Position Analysis, and Smaller Conversion Budgets (Q2–Q3)

2. Sensitivity to Spatial Block Ordering and Global Spatial Reasoning (Q2)

Direct Answer: Reversing the block sequence or adopting a serpentine scan changes the 12-benchmark mean by 0.3–0.4 points (74.0 vs. 73.6 and 73.7), with changes of −1.1-1.1 to +0.4+0.4 points on the reported document, chart, and spatial reasoning benchmarks. In a floating-point DocVQA analysis at a fixed 1344×13441344 \times 1344 resolution, grouped by answer position, original-order BLoCE remains within 0.4 points of full attention across early, middle, and late positions; reversing the block order changes each group's score by at most 0.7 points.

Evidence and Analysis: We evaluated two alternative block traversal orders using the same image-resolution settings as AndesVL, keeping token ordering within each block, preserving the original 2D positional embeddings, and restoring the canonical 2D token order before the projector:

  1. Reversed block order (bottom-to-top): Reversing the block sequence {K,K−1,…,1}\{K, K-1, \dots, 1\}.
  2. Serpentine traversal: Alternating horizontal scan directions across spatial rows of blocks.
Block Traversal Order12-Benchmark MeanDocVQAInfoVQAChartQARealWorldQAMMT-Bench
Original order (top-to-bottom)74.094.278.986.571.063.9
Reversed block order (bottom-to-top)73.693.6 (−0.6)79.2 (+0.3)85.8 (−0.7)70.2 (−0.8)63.7 (−0.2)
Serpentine order (alternating rows)73.793.5 (−0.7)78.6 (−0.3)86.9 (+0.4)69.9 (−1.1)63.3 (−0.6)
  1. Task-Level Stability: Across the five representative benchmarks requiring dense text reading, chart alignment, and global spatial reasoning, both alternative traversal orders yield small variations within [−1.1,+0.4][-1.1, +0.4] points. Within the vision encoder, each block retains bidirectional attention over B=128B=128 tokens with 2D positional embeddings while the selected KV history supplies context from preceding blocks; retaining all output tokens in their restored 2D order for the downstream LLM may also help explain the observed task performance across traversal orders.
  2. Answer-Position Analysis: On DocVQA in floating point at a fixed 1344×13441344 \times 1344 resolution, we group questions by the block containing the answer-region center in the original token scan and keep group membership fixed across runs. Original-order BLoCE scores 94.0 / 94.0 / 93.6 for early / middle / late answers, compared with 94.1 / 93.6 / 93.8 for full attention (differences of −0.1 / +0.4 / −0.2 points). Reversing the block order gives 93.8 / 93.3 / 93.1 (−0.2 / −0.7 / −0.5 points vs. original order), so original-order BLoCE scores higher in all three groups (full table in AubG Q3).

3. Performance with Smaller Conversion Datasets and Shorter Training Schedules (Q3)

Direct Answer: Reducing the conversion budget to 20% (12.2 hours / 390 GPU-hours) already recovers 68% of the accuracy drop from BC-direct (70.1 →\to 72.7), and a 50% budget (30.2 hours / 966 GPU-hours) reaches 73.4 (within 0.5 points of the source encoder's 73.9).

Evidence and Analysis: Progressive Conversion trains for a single pass over the 5.7M post-training dataset (179,200 steps×batch size 32179,200\text{ steps} \times \text{batch size } 32, Table 10). The reduced-data runs in Table 5 use proportionally fewer update steps while completing all progressive stages. The table below reports measured wall-clock training time on 32 H20 GPUs (GPU-hours = wall-clock hours ×32\times 32, rounded to the nearest integer):

Conversion Budget (Table 5)Training SamplesTraining StepsWall-Clock (32 H20s)Compute Cost12-Benchmark Mean
0% (BC-direct, no training)000 h0 GPU-hours70.1
20% Budget1.14M35,84012.2 h390 GPU-hours72.7
50% Budget2.85M89,60030.2 h966 GPU-hours73.4
100% Full Conversion (BLoCE)5.70M179,20060.4 h1933 GPU-hours74.0

Conversion cost: BLoCE is designed to reuse an existing pretrained MLLM vision encoder through attention-only fine-tuning, avoiding the cost of training a new large vision encoder from scratch. The full conversion uses at most 5.7M post-training samples—approximately 0.05% of the 12 billion image-text samples reported for AIMv2 pretraining (Section 4.1)—to adapt a pretrained MLLM encoder into a mobile-friendly one, while the 20% budget (390 GPU-hours) offers a practical operating point when compute resources are constrained.

I particularly appreciate the comparisons with tiled and window-based attention, the historical KV cache ablations, and the successful conversion of a bidirectionally pretrained SigLIP2 encoder. These results substantially strengthen the methodological justification and generalizability of BLoCE. The new device memory, memory traffic, and end-to-end latency measurements also reinforce the practical significance of the approach. While cross-vendor hardware validation and energy measurements remain open limitations, I believe the rebuttal has convincingly addressed the main concerns raised during the discussion. I therefore update my recommendation from Weak Accept to Accept.

AuthorsOpenReview ↗

Sincere Appreciation to Reviewer wFyu

We sincerely thank the reviewer for the thoughtful reassessment and updated recommendation. We greatly appreciate your constructive feedback, which has helped us further strengthen our work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response : Summary of New Experiments, Device Measurements, and Core Clarifications (Updated with Cross-Vendor Validation)

We sincerely thank all reviewers (wFyu, AubG, 6ARt) for their constructive reviews and for recognizing BLoCE's practical motivation, deployment-friendly operator preservation, and real-device NPU acceleration. Below we summarize our new experiments, device measurements, and clarifications addressing the core questions across the reviews:

  1. Generalization to a Bidirectionally Pretrained Vision Encoder (6ARt Q1): We applied Progressive Conversion to the SigLIP2-derived 300M vision encoder initialized from Qwen3-VL-4B-Instruct, preserving its native patch embedding and positional encoding while keeping the merger/projector and LLM frozen across all five compared settings. With the same conversion dataset and training budget as our AIMv2 experiments, direct block-causal masking (BC-direct) lowers the 12-benchmark mean from 71.3 to 66.9. Progressive Conversion recovers it to 71.5 with full history (BC-conv) and 71.1 with bounded history (BLoCE), 0.2 points below the full-attention reference and 0.5 below same-budget FA-tuned (71.6). Across all 12 benchmarks, BLoCE improves over BC-direct by 3.1–5.1 points (within −0.7 to +0.9 points of the full-attention reference), demonstrating substantial conversion recovery on a bidirectionally pretrained encoder beyond AIMv2.
  2. Comparison with Tiled Encoding and ViTDet-Style Attention on the Same Backbone (AubG Q1, 6ARt Q2): Under the same 5.7M conversion budget at 1344×13441344 \times 1344 on the Dimensity 9500 NPU (Table 15 joint quantized protocol), BLoCE (71.0 mean, 699 ms, 9,216 tokens) is 0.4 points below Source AIMv2 (71.4) and 0.2 points above the same-data FA-tuned control (70.8). It achieves higher quality and lower latency than all four tiled variants: 2×22\times2 tiles (69.3 / 1381 ms), 3×33\times3 tiles (68.6 / 1103 ms), 2×22\times2 tiles + thumbnail (69.4 / 1732 ms), and 3×33\times3 tiles + thumbnail (68.8 / 1224 ms). The ViTDet-style configuration (16×1616\times16 local windows in 20 layers and global attention in layers 6, 12, 18, and 24) obtains 70.4 / 1208 ms. Pure local windows are faster (641 ms) but score 66.9; BLoCE gains 4.1 points for 58 ms (9.0%) additional encoder latency.
  3. Ablation of Historical KV Context and Selection (AubG Q2): At B=128B=128, removing cross-block history (C = 0) lowers the 12-benchmark mean from 74.0 to 67.6 (−6.4-6.4 points), keeping only the immediately preceding block (C = 128) yields 70.1, and simple recency (C = 256) yields 71.4. Our importance–redundancy selection (C = 256, retaining W=128W=128 recent entries plus 128 entries selected from M≤C+B−W=256M \le C+B-W = 256 older candidates) reaches 74.0, within 0.1 points of full history (74.1).
  4. Peak Memory, Memory Traffic, and End-to-End TTFT (wFyu Q1, AubG Q4, 6ARt Q3): At 1344×13441344 \times 1344, BLoCE reduces end-to-end MLLM Time-to-First-Token (TTFT, covering vision encoding, projector, and LLM prefill) from 4231 ms to 2739 ms (1.54× speedup). On the Dimensity 9500 NPU (vision encoder only), BLoCE reduces peak device memory from 3.50 GiB to 1.08 GiB (−69.1%) and lowers external-memory traffic by 54%. Within the 699 ms encoder latency, the 24-layer block forward passes take 648 ms, and historical KV scoring, selection, data movement, and cache updates take 51 ms (∼0.71 ms/block\sim 0.71\text{ ms/block}).
  5. Cross-Vendor Applicability (wFyu Weakness 1, AubG Weakness 3): Encoder latency falls from 2413 to 814 ms (2.96×) on MediaTek Dimensity 9400 (NeuroPilot) and from 2814 to 1053 ms (2.67×) on Qualcomm Snapdragon SM8850 Hexagon (llama.cpp), using W8 weights and B=128,C=256B=128, C=256 at 1344×13441344 \times 1344. Each speedup is evaluated against the full-attention baseline on the same platform and toolchain. These results support applicability across three NPU platforms, two vendors, and two toolchains.
  6. Spatial Traversal Orders, Answer Positions, and Smaller Conversion Budgets (wFyu Q2–Q3, AubG Q3): Reversing the block traversal order (bottom-to-top) or using a serpentine traversal changes the 12-benchmark mean by 0.3–0.40.3\text{--}0.4 points (73.6 and 73.7 vs. 74.0), with variations between −1.1-1.1 and +0.4+0.4 points across representative high-resolution document, chart, and spatial reasoning benchmarks. A floating-point DocVQA analysis at a fixed 1344×13441344 \times 1344 resolution, grouped by answer position, shows that original-order BLoCE remains within 0.4 points of full attention across early, middle, and late positions (−0.1 / +0.4 / −0.2 points), while reversing the block order changes the scores by −0.2 / −0.7 / −0.5 points on the same groups. Furthermore, 20% and 50% conversion budgets (Table 5) take 12.2 hours (390 GPU-hours) and 30.2 hours (966 GPU-hours) in measured training time while recovering the 12-benchmark mean from 70.1 to 72.7 and 73.4.

We have incorporated the new results and discussions into the revised manuscript.

AuthorsOpenReview ↗

Final remarks on the Rebuttal and Revision Process with Sincere Appreciation to Reviewers and the Area Chair

We thank the Area Chair and all reviewers for their time, careful reading, and constructive discussion.

  • Strengths: The reviewers recognized BLoCE's practical motivation (wFyu, AubG, 6ARt), deployment-friendly design (wFyu, AubG, 6ARt), effective conversion of pretrained encoders (wFyu, 6ARt), and substantial real-device acceleration with comparable downstream performance (wFyu, AubG, 6ARt).
  • Concerns: Reviewers wFyu and 6ARt confirmed that their main concerns were addressed. Reviewer AubG acknowledged that most of their concerns had been resolved, with cross-vendor applicability as the sole remaining concern; we subsequently provided additional cross-vendor experiments on Qualcomm Hexagon demonstrating a 2.67× encoder speedup and are awaiting their final feedback.
  • Revisions: We expanded the experiments and analyses, placing the tiled/windowed comparisons and SigLIP2 conversion results in the main text, with the remaining additions in the supplementary material.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.