acceptodds
Under review as a conference paper at ICLR 2027

FastVLM2: Efficient On-Device Dense Grounding for Generalist VLMs

Abstract

Generalist vision-language models (VLMs) with dense grounding capabilities can perform a broad range of classical vision tasks through a single model and natural-language interface. However, efficient on-device inference with such models faces two complementary bottlenecks: costly high-resolution visual encoding and lengthy autoregressive decoding of numerous spatial locations. We introduce FastVLM2, a unified framework that addresses both. For input-side efficiency, we introduce a native-resolution strategy that adapts pretrained convolutional and hybrid convolution-transformer backbones (ConvNeXt and FastViTHD, respectively) through VLM instruction tuning, enabling arbitrary image resolutions and aspect ratios without additional vision pretraining, image tiling, or explicit layout tokens. On a MacBook with an M1 Max, FastVLM2-4B achieves VQA accuracy comparable to Qwen3-VL-4B with lower prefill latency. For output-side efficiency, we use quantized location tokens to compactly represent grounding outputs, reducing the output-token budget by over relative to textual coordinate representations. To recover the localization accuracy lost to quantization, we introduce a simple Gaussian distance-aware soft cross-entropy objective that brings quantized grounding close to textual-coordinate performance while preserving this decoding advantage. At higher grounding accuracy, FastVLM2-1.8B achieves lower end-to-end latency than Rex-Omni. We further introduce a training-free, geometry-aware box confidence estimator that directly exploits the predictive distributions of quantized location tokens. We show that this scoring rule generalizes across three independently trained grounding models and consistently improves COCO mAP by more than points over the score-free ranking protocol used in Rex-Omni. Together, FastVLM2 makes accurate, confidence-aware dense grounding practical for on-device generalist VLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Efficient Inference, Visual Grounding, Native-Resolution Vision, Structured Prediction, Confidence Estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NchQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FastVLM2 extends FastVLM to support native-resolution inputs and efficient dense visual grounding. It adapts pretrained hierarchical vision encoders through lightweight positional modifications during VLM fine-tuning, without additional vision pretraining or image tiling. For grounding, it uses compact quantized location tokens, a Gaussian distance-aware loss to improve localization, and a training-free, geometry-aware score for ranking predicted boxes. The paper validates these components through encoder and loss ablations, VQA and grounding benchmarks, and latency measurements on an M1 Max MacBook. It also applies the box-scoring method to two independently trained grounding models, where it improves confidence-ranked COCO mAP.

Critical strengths
  1. A strong efficiency–accuracy trade-off. FastVLM2-4B achieves nearly the same average VQA score as Qwen3-VL-4B (81.7 vs. 82.1) with substantially lower prefill latency on the tested device (1.2 vs. 5.6 seconds), while performing better on the reported grounding averages. This supports the paper’s practical case for a generalist on-device model.

  2. A transferable box-confidence method. The training-free geometry-aware score improves COCO mAP by 8.3–8.6 points over score-free ranking across three independently trained models. Because it improves the ranking of unchanged predicted boxes without retraining, the result supports a contribution beyond FastVLM2’s particular model architecture.

Critical weaknesses
  1. Limited methodological novelty. FastVLM2 retains the overall FastVLM architecture and combines existing positional mechanisms with quantized coordinates already used by Rex-Omni. Its grounding loss and confidence score are useful refinements, but the paper’s main contribution appears to be an effective system integration rather than a substantial new modeling principle.

  2. Incomplete attribution of the final gains. The native-resolution ablations in Section 4.1 use Qwen2.5-1.5B, and the grounding-loss ablations in Section 4.2 use Qwen2.5-3B; the main results use Qwen3 decoders after SFT and RL. Without a matched baseline under the final training recipe, it is difficult to determine how much of the reported accuracy advantage comes from the proposed components rather than the changed model and training setup. This weakens the evidence for the paper’s central method claims.

Questions for authors
  1. Could the authors provide the key component ablations using the Qwen3 decoder employed in the final FastVLM2 models? The current ablations use Qwen2.5, so it is unclear whether the observed gains carry over to the Qwen3-based models reported in the main results.

2.Interpretation of latency: Appendix G measures decoding with a fixed seven-box budget, while the headline 7.3× comparison uses selected accuracy–resolution operating points. How representative is that speedup of end-to-end latency when models generate their natural, variable-length outputs on the grounding benchmark?

3 comments
AuthorsOpenReview ↗

Response to Reviewer NchQ - Part 1 / 2

We thank the reviewer for recognizing FastVLM2's accuracy–latency trade-off and transferable confidence estimator. Below, we clarify our methodological contributions and present new Qwen3-based ablations and end-to-end on-device measurements with natural-length grounding outputs.

1. “Limited methodological novelty.” (W1)

We build on FastVLM and do not claim CoordConv, 2D RoPE, or quantized coordinates as individually new. Our contributions address three complementary challenges in efficient generalist VLMs:

Native-resolution hierarchical encoders. To the best of our knowledge, FastVLM2 provides the first fine-tuning-only recipe for adapting pretrained hierarchical convolutional (ConvNeXt) and hybrid (FastViTHD) encoders to variable native resolutions and aspect ratios within a single checkpoint, without additional vision pretraining, tiling, or explicit layout tokens. Unlike fixed-resolution FastVLM checkpoints, this enables multiple accuracy–latency operating points from the same model. The recipe combines lightweight positional adaptations with a staged training curriculum.

Geometry-aware supervision for compact location tokens. Gaussian distance-aware supervision substantially improves localization accuracy when training with quantized location tokens, mitigating the accuracy loss from quantization while preserving the decoding efficiency of compact outputs (over 60% fewer tokens).

Transferable, training-free box confidence. Our geometry-aware estimator uses existing location-token distributions to score predicted boxes. It requires no additional training, confidence head, or forward passes and generalizes to other models using compact location tokens, improving COCO mAP by over 8 points across three independently trained models.

2. “Could the authors provide the key component ablations using the Qwen3 decoder employed in the final FastVLM2 models?” (Q1) “Incomplete attribution of the final gains.” (W2)

We performed a controlled Qwen3-4B grounding-loss ablation using the final recipe's full Stage 1.0/1.5/2.0 schedules (5k/38k/33k steps), rather than the shorter Qwen2.5 ablations in the submission. The runs differ only in their grounding objective: NTP-only versus Gaussian distance-aware supervision.

Training objectiveCOCO F1 @ IoU 0.5 ↑LVIS F1 @ IoU 0.5 ↑Avg. referring expressions ↑
NTP-only (no auxiliary loss)57.152.668.3
Gaussian DA-SoftCE (ours)60.055.269.4
Δ (ours − NTP-only)+2.9+2.6+1.1

Gaussian supervision improves both dense grounding and referring expressions with the Qwen3-4B decoder, supporting the transfer of this contribution to the final decoder family and longer training schedule. This experiment isolates the loss contribution through Stage 2.0.

AuthorsOpenReview ↗

Response to Reviewer NchQ - Part 2 / 2

3. “How representative is that speedup of end-to-end latency when models generate their natural, variable-length outputs on the grounding benchmark?” (Q2)

Appendix G's fixed-seven-box evaluation controls decoding length, while the reported 7.3× FastVLM2-1.8B versus Rex-Omni-3B comparison uses selected accuracy–resolution operating points. To assess realistic inference, we benchmarked FastVLM2-4B, Qwen3-VL-4B, and Rex-Omni-3B on the same 100 COCO validation images (Apple M1 Max, MLX; 4-bit LLM, FP16 vision encoder). Measurements include preprocessing, vision encoding, prefill, natural-length decoding, and postprocessing, including confidence scoring. This precision preserves FastVLM2 grounding accuracy (73.4 vs. 73.6 F1 @ IoU 0.5 with full FP16).

Matched 100-image COCO subsetQwen3-VL-4BRex-Omni-3BFastVLM2-4B (ours)
Mean end-to-end latency (s) ↓7.1681.1500.775
Median end-to-end latency (s) ↓1.9971.0730.632
p90 / p99 latency (s) ↓37.85 / 47.111.59 / 2.461.23 / 2.09
Mean prefill (s) ↓0.5620.6740.319
Mean decoding (s) ↓6.5900.4570.429
Mean visual tokens ↓271357153
Mean output tokens ↓60453.144.5
FastVLM2 improvement overMean end-to-endMedian end-to-endPrefill
Qwen3-VL-4B9.25× faster3.16× faster1.76× faster
Rex-Omni-3B1.48× faster1.70× faster2.11× faster

Qwen3-VL generates much longer outputs (604 vs. 44.5 mean tokens), with 10/100 examples reaching the generation cap; we therefore report median and tail latencies rather than attributing the full mean speedup to vision encoding. Rex-Omni is the more informative token-format comparison: it also uses compact locations and generates similar-length outputs (53.1 vs. 44.5 tokens), yet FastVLM2 achieves 2.11× faster prefill and 1.48× faster mean end-to-end inference, despite its larger decoder (4B vs. 3B).

Confidence estimation adds approximately 0.5 ms total for seven boxes (28 coordinates), already included in end-to-end measurements. FastVLM2 also uses 29.4 J/image and 3.47 GB peak MLX memory, versus 241.4 J / 5.42 GB for Qwen3-VL and 44.6 J / 4.04 GB for Rex-Omni.

These results show that FastVLM2's efficiency advantage persists with natural-length grounding outputs, while the Qwen3 ablation supports the proposed loss. We will incorporate these measurements and distinguish the fixed-budget, selected-operating-point, and natural-output evaluations in the revised paper.

Reviewer_NchQOpenReview ↗

Maintain Weak Acceptance

Thank you for the clarifications and additional experiments. The Qwen3-4B ablation and natural-length end-to-end latency measurements substantially address my main empirical concerns. The comparison with Rex-Omni is particularly informative, showing an efficiency advantage with similar output lengths. Please clearly distinguish these results from the original 7.3× comparison in the revised paper.

While my reservation about methodological novelty remains, I recognize the paper’s practical value and maintain my rating of 3 (Weak acceptance).

Reviewer_qrjPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FastVLM2, an efficient generalist VLM supporting dense visual grounding. It addresses two inference bottlenecks: high-resolution image encoding and autoregressive coordinate generation. The proposed approach combines native-resolution adaptation of pretrained hierarchical vision encoders using CoordConv and 2D RoPE, Gaussian distance-aware supervision for quantized location tokens, and training-free, geometry-aware bounding-box confidence estimation. On an M1 Max, FastVLM2-4B achieves comparable average VQA performance to Qwen3-VL-4B with approximately 4.6× lower prefill latency.

Critical strengths
  1. The paper demonstrates compelling accuracy-latency trade-offs across multiple model sizes and grounding benchmarks, with comparisons conducted on the same hardware and inference framework.

  2. Gaussian distance-aware soft cross-entropy improves RefCOCO average accuracy from 71.9 to 76.4, substantially reducing the gap between quantized and textual coordinates.

  3. Transferable confidence estimation. The geometry-aware confidence estimator improves COCO mAP across three independently trained models without additional training. Comparisons against alternative scoring methods strengthen this contribution.

Critical weaknesses
  1. Limited novelty of individual techniques. CoordConv, 2D RoPE, quantized coordinate tokens, and soft-label supervision are established techniques. The contribution is primarily their effective integration and empirical validation rather than a fundamentally new methodology.

  2. Insufficient isolation of final-model improvements. The main results use a substantially extended five-stage SFT pipeline followed by RL, whereas the architectural and grounding-loss ablations use shorter training schedules and different LLM backbones. Consequently, the extent to which individual proposed techniques explain the final performance advantage remains unclear.

  3. Latency evaluation does not fully capture realistic deployment. Real workload latency, confidence-scoring overhead, peak memory consumption, and energy efficiency are not characterized. These omissions weaken the broader on-device efficiency claims.

Questions for authors
  1. Can the authors provide final-model ablations isolating native-resolution adaptation and Gaussian supervision under the same extended SFT and RL pipeline?

  2. How do the latency advantages change when evaluating actual generated grounding sequences, including confidence estimation and postprocessing?

3 comments
AuthorsOpenReview ↗

Response to Reviewer qrjP - Part 1 / 2

We thank the reviewer for recognizing FastVLM2's accuracy–latency trade-offs, the benefits of Gaussian supervision, and the transferability of our confidence estimator. Below, we clarify our methodological contributions and present new Qwen3 ablations and natural-length on-device measurements.

1. “Limited novelty of individual techniques.” (W1)

We agree that CoordConv, 2D RoPE, and quantized location tokens are established components. Our contributions are their adaptation to efficient native-resolution encoding, improved supervision for compact grounding, and a transferable confidence estimator. These methods together with the proposed learning techniques gives us state-of-the-art efficiency-accuracy tradeoff for generalist VLM with grounding capability.

Native-resolution hierarchical encoders. To the best of our knowledge, FastVLM2 is the first fine-tuning-only recipe to adapt pretrained hierarchical convolutional (ConvNeXt) and hybrid (FastViTHD) encoders to native resolutions and aspect ratios within one checkpoint, without additional vision pretraining, tiling, or explicit layout tokens. Unlike fixed-resolution FastVLM checkpoints, this enables multiple accuracy–latency operating points from the same model. The recipe combines CoordConv, 2D RoPE for hybrid encoders, and a staged curriculum.

Geometry-aware supervision for compact location tokens. Gaussian distance-aware supervision substantially improves localization accuracy when training with quantized location tokens, mitigating the accuracy loss from quantization while preserving the decoding efficiency of compact outputs (over 60% fewer tokens).

Transferable, training-free box confidence. Our geometry-aware estimator uses existing location-token distributions to score box geometry. It requires no additional training, confidence head, or forward passes and generalizes across independently trained models using compact location tokens, improving COCO mAP by approximately 8–9 points.

2. “Can the authors provide final-model ablations isolating native-resolution adaptation and Gaussian supervision under the same extended SFT and RL pipeline?” (Q1) “Insufficient isolation of final-model improvements.” (W2)

Full five-stage SFT + RL ablations for every component are prohibitively expensive during rebuttal. To strengthen attribution beyond the submission's shorter ablations, we performed a controlled Qwen3-4B scale comparison using the final recipe's full Stage 1.0/1.5/2.0 schedules (5k/38k/33k steps). Only the grounding loss differs: NTP-only versus Gaussian DA-SoftCE.

Training objectiveCOCO F1 @ IoU 0.5 ↑LVIS F1 @ IoU 0.5 ↑Avg. referring expressions ↑
NTP-only (no auxiliary loss)57.152.668.3
Gaussian DA-SoftCE (ours)60.055.269.4
Δ (ours − NTP-only)+2.9+2.6+1.1

Gaussian supervision improves grounding at the Qwen3-4B scale, with +2.9 COCO F1, +2.6 LVIS F1, and +1.1 referring-expression accuracy. This directly tests transfer to the final decoder family and the longer Stage-2.0 schedule.

We additionally trained a FastVLM baseline with the same Qwen3-0.6B decoder, data mixture, and training schedule as FastVLM2. Under these matched conditions, FastVLM2 achieves +7.2 COCO mAP and 3.28× faster end-to-end inference on dense grounding task, demonstrating improvements beyond differences in training data or schedule. Please see our response to Reviewer 1xbU (W3/Q1) for detailed results and analysis.

AuthorsOpenReview ↗

Response to Reviewer qrjP - Part 2 / 2

3. “How do the latency advantages change when evaluating actual generated grounding sequences, including confidence estimation and postprocessing?” (Q2) “Latency evaluation does not fully capture realistic deployment.” (W3)

We evaluated Qwen3-VL-4B, Rex-Omni-3B, and FastVLM2-4B on the same 100 COCO validation images (Apple M1 Max Macbook Pro, MLX; 4-bit LLM, FP16 vision encoder), measuring preprocessing, vision encoding, prefill, natural-length decoding, and postprocessing, including FastVLM2 confidence scoring. This precision preserves FastVLM2 grounding accuracy (73.4 vs. 73.6 F1 @ IoU 0.5 with full FP16).

Matched COCO subsetQwen3-VL-4BRex-Omni-3BFastVLM2-4B (ours)
Mean end-to-end latency (s) ↓7.1681.1500.775
Median end-to-end latency (s) ↓1.9971.0730.632
p90 / p99 end-to-end (s) ↓37.85 / 47.111.59 / 2.461.23 / 2.09
Mean prefill (s) ↓0.5620.6740.319
Mean decoding (s) ↓6.5900.4570.429
Mean visual tokens ↓271357153
Mean output tokens ↓60453.144.5
CPU+GPU energy (J/image) ↓241.444.629.4
Peak MLX memory (GB) ↓5.424.043.47
FastVLM2 improvement relative toMean E2EMedian E2EPrefillEnergyPeak memory
Qwen3-VL-4B9.25× faster3.16× faster1.76× faster8.21× lower1.56× lower
Rex-Omni-3B1.48× faster1.70× faster2.11× faster1.52× lower1.16× lower

Qwen3-VL generates substantially longer outputs (604 vs. 44.5 tokens), with 10/100 examples reaching its generation cap. We therefore include median and tail latencies rather than attributing the full mean speedup to vision encoding. Rex-Omni also uses compact location tokens and produces similar-length outputs (53.1 vs. 44.5 tokens); FastVLM2 still achieves 2.11× faster prefill and 1.48× faster mean end-to-end inference, despite using a larger decoder (4B vs. 3B).

Confidence scoring costs approximately 0.5 ms total for seven boxes (28 coordinates), already included in end-to-end latency along with box parsing. Energy measurements use zeus-apple-silicon with Apple IOReport counters.

We also measured natural-length VQA on 500 matched examples per benchmark:

BenchmarkQwen3-VL-4B (s) ↓FastVLM2-4B (ours, s) ↓Speedup ↑
DocVQA4.361.133.87×
InfoVQA3.431.033.32×
ChartQA0.950.551.72×
OCRBench1.440.492.96×

These new measurements demonstrate lower end-to-end latency, memory use, and energy consumption with natural outputs, while the larger-scale Qwen3 ablation supports the grounding-loss contribution. We will incorporate the new results and clarify the evaluation protocols in the revised paper.

AuthorsOpenReview ↗

Follow-up on Rebuttal

Dear Reviewer qrjP,

Thank you again for your thoughtful review and constructive feedback.

We have provided a detailed response to your concerns, including additional Qwen3-based ablations and end-to-end on-device evaluations covering latency, memory, and energy efficiency.

We would appreciate it if you could review our response and let us know whether it addresses your concerns. We would be happy to provide any further clarifications.

Thank you for your time and consideration.

Reviewer_1xbUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

FastVLM2 aims to make generalist VLMs faster on local devices. It combines native-resolution visual encoding, Gaussian supervision for quantized coordinates, and a training-free confidence score. The results show strong performance and lower latency on visual question answering and grounding tasks.

Critical strengths
  1. The problem has clear practical value. The model achieves strong results with substantial speed improvements under the tested setup.

  2. The confidence score is interesting and requires no extra training or model forward passes. Its improvements across three models are encouraging.

  3. The paper includes extensive ablations on positional encoding, training schedules, coordinate initialization, losses, and confidence scoring.

Critical weaknesses
  1. The new contributions need a clearer comparison with prior work. FastViTHD comes from FastVLM [1]. ConvLLaVA [2] already studies hierarchical encoders and varying image shapes. Florence-2 [3] uses unified task generation and quantized location tokens. The main additions are the adaptation recipe, Gaussian loss, and confidence score. Their advantages over related methods need to be clearer.

  2. Important baselines are missing. FastVLM is the most relevant baseline because the paper reuses its backbone. ConvLLaVA and NativeRes-LLaVA [4] would also provide useful comparisons. Florence-2 could be tested on shared grounding tasks, and MobileVLM V2 [5] could serve as a lightweight VQA reference.

  3. The effect of training data is unclear. The local ablations use controlled settings, but there is no closely related baseline trained on the same data and schedule. Later stages reuse grounding data and change the task mixture, without measuring these effects separately. Per-source sample counts and sampling ratios are also not fully reported.

[1] Vasu et al. FastVLM: Efficient Vision Encoding for Vision Language Models. CVPR, 2025. [2] Ge et al. ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models. arXiv, 2024. [3] Xiao et al. Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks. CVPR, 2024. [4] Niu et al. Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models. arXiv, 2025. [5] Chu et al. MobileVLM V2: Faster and Stronger Baseline for Vision Language Model. arXiv, 2024.

Questions for authors
  1. Could you train a FastVLM-style baseline with the same LLM, data mixture, and schedule, and measure the separate effects of additional grounding data and repeated exposure?

  2. Table 1b uses the original positional encoding. Does introducing native-resolution training earlier still hurt after adding CoordConv and 2D RoPE? How does fixed-resolution training perform with the same positional encodings?

  3. Table 3 suggests that the confidence score works across models. How do its gains change with the training loss? Could you report the mAP gain from scoring for NTP-only training and Gaussian supervision with different kernel widths and mixing weights?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1xbU - Part 1 / 3

We thank the reviewer for highlighting the practical value, confidence estimator, and extensive ablations. We address the reviewer’s concerns by clarifying our contributions relative to prior work and presenting new experiments with matched baselines, on-device measurements, native-resolution training curriculum, and loss ablations.

1. “The new contributions need a clearer comparison with prior work.” (W1)

FastVLM established efficient hierarchical visual encoding, however it uses a fixed input resolution, resizing and padding images to a predefined square resolution (1024 × 1024), regardless of their original dimensions or aspect ratios. FastVLM2 adapts pretrained hierarchical encoders to variable native resolutions and aspect ratios within one checkpoint, without additional vision pretraining. Our contributions are: (i) a lightweight adaptation and curriculum for hybrid (FastViTHD) and convolutional (ConvNeXt) encoders; (ii) Gaussian distance-aware supervision that improves localization with compact quantized coordinates; and (iii) training-free, geometry-aware box confidence estimator that generalizes across models using compact location tokens.

ConvLLaVA explores hierarchical vision encoders and supports varying image aspect ratios, but requires separate checkpoints for each target resolution (768², 1024², or 1536²). Each checkpoint operates at a fixed resolution, with images resized and padded to fit, unlike FastVLM2, which supports multiple resolutions and aspect ratios within a single checkpoint. Florence-2 uses quantized location tokens, which we do not claim as novel. Our contributions focus on geometry-aware supervision to improve localization accuracy when training models with compact location tokens, and a training-free confidence scoring mechanism that generalizes across models using such tokens. Furthermore, unlike Florence-2, FastVLM2 is a generalist VLM that supports both conversational VQA and grounding within a single model.

2. “Could you train a FastVLM-style baseline with the same LLM, data mixture, and schedule, and measure the separate effects of additional grounding data and repeated exposure?” (Q1) “The effect of training data is unclear.” (W3)

We trained a FastVLM baseline using the same Qwen3-0.6B LLM, data mixture, and training schedule as FastVLM2. Despite using significantly fewer tokens, FastVLM2 achieves comparable or better accuracy across VQA and grounding tasks while being faster (e.g., a 3.28× mean end-to-end speedup on COCO). These results demonstrate that the proposed native-resolution encoding and grounding tokens improve the accuracy–efficiency trade-off.

ModelAvg. VQA ↑Avg. referring expressions ↑Avg. dense grounding ↑mAP on COCO ↑
FastVLM-baseline, fixed resolution - 1024²73.078.349.337.1
FastVLM2, native resolution (ours)73.477.051.044.3
Δ (ours − fixed)+0.4−1.3+1.7+7.2

With training matched, FastVLM2 improves Avg. VQA by +0.4, dense grounding by +1.7, and COCO mAP by +7.2, while referring-expression accuracy is 1.3 points lower at native resolution. FastVLM upscales the typically lower-resolution RefCOCO inputs to 1024²; at a matched visual-token budget, FastVLM2 increases from 77.0 to 78.2 (vs. FastVLM's 78.3), without retraining.

On-device comparison. Both models (FastVLM-baseline and FastVLM2) using the Qwen3-0.6B LLM were evaluated on the same 500 COCO val2017 images (Apple M1 Max Macbook Pro, MLX, 4-bit LLM, FP16 vision tower); end-to-end timing includes preprocessing, prefill, natural-length decoding, and postprocessing. CPU+GPU energy is also measured. We independently verified that the quantized setup preserves grounding accuracy (73.4 vs. 73.6 F1@IoU 0.5 for the full-FP16 FastVLM2-4B checkpoint).

FastVLM-baseline / FastVLM2 settingMean visual tokens ↓Prefill (s) ↓Mean end-to-end (s) ↓Output tokens ↓Energy (J/image) ↓
Fixed 1024² + textual coordinates2560.1591.082242.225.75
FastVLM2 native + quantized <loc_K> (ours)1510.1100.33054.08.33
Improvement (baseline / ours)1.70× fewer1.45× faster3.28× faster4.49× fewer3.09× lower

On these COCO images (276k pixels on average), native processing reduces visual tokens from 256 to 151 and prefill time from 0.159 s to 0.110 s. Compact location tokens also reduce output tokens from 242.2 to 54.0, contributing to 3.28× lower end-to-end latency and 3.09× lower energy. These gains reflect both native-resolution processing and compact outputs, not a universal speedup at every input size.

AuthorsOpenReview ↗

Response to Reviewer 1xbU - Part 2 / 3

3. “Important baselines are missing.” (W2)

The matched FastVLM comparison is above. We additionally compare NativeRes-LLaVA-1.5B and MobileVLM V2-1.7B with FastVLM2-1.8B. Accuracy is evaluated using lmms-eval (DocVQA/InfoVQA: ANLS; ChartQA: relaxed accuracy; OCRBench: accuracy). End-to-end latency is measured on 500 matched samples per benchmark on Apple M1 Max MacBook Pro/MLX (4-bit LLM, FP16 vision encoder; natural-length generation). FastVLM2 uses native inputs capped at 1536² for the NativeRes-LLaVA comparison. We use 336² inputs for FastVLM2 to match MobileVLM V2’s input resolution, ensuring a fair comparison at the same visual input resolution.

Baseline / input settingBenchmarkBaseline accuracy ↑FastVLM2 accuracy ↑Baseline latency (s) ↓FastVLM2 latency (s) ↓Speedup ↑
NativeRes-LLaVA-1.5B / 1536² capDocVQA81.8596.33.670.695.35×
InfoVQA40.0783.62.820.634.50×
ChartQA70.2482.20.820.352.39×
OCRBench61.5072.40.840.292.90×
MobileVLM V2-1.7B / 336²DocVQA14.9620.00.2250.1451.55×
InfoVQA14.0824.20.2370.1451.63×
ChartQA12.5233.50.1870.1081.74×
OCRBench24.7040.10.3010.1571.92×

FastVLM2 is more accurate on all four benchmarks at the corresponding input settings, with 2.39–5.35× lower latency than NativeRes-LLaVA and 1.55–1.92× lower latency than MobileVLM V2.

Florence-2-large (0.77B) vs. FastVLM2-0.7B, COCO at 768²:

MetricFlorence-2-largeFastVLM2-0.7B (ours)Δ / speedup
COCO F1 @ IoU 0.5 ↑73.873.6−0.2 points
Mean end-to-end latency (s) ↓0.5970.2932.04× faster
Energy (J/image) ↓17.07.72.21× lower

At 768², FastVLM2 matches Florence-2's COCO grounding F1 to within 0.2 points, while achieving 2.04× lower latency and 2.21× lower energy, with the additional ability to perform conversational VQA. ConvLLaVA's reported checkpoints use a 7B decoder, precluding a close size-matched comparison.

4. “Does introducing native-resolution training earlier still hurt after adding CoordConv and 2D RoPE? How does fixed-resolution training perform with the same positional encodings?” (Q2)

The following shorter three-stage schedule ablations extend Table 1(b):

Positional encodingStage 1.5Stage 2.0Avg.5 ↑
Original (no additional PE)Fixed res.Native res.61.4
Original (no additional PE)Native res.Native res.44.2
CoordConv + 2D RoPENative res.Native res.44.5
CoordConv + 2D RoPE (our curriculum)Fixed res.Native res.66.8
Δ vs. early-native, same PE+22.3

Early native-resolution training still hurts with CoordConv + 2D RoPE (44.5 vs. 66.8); the added encodings improve early-native training only +0.3 points. With the preferred curriculum, they improve Avg.5 by +5.4 (61.4 → 66.8). One possible explanation is the difficulty of learning newly introduced, non-pretrained CoordConv layers alongside varying image sizes.

Remaining control: A fully fixed-resolution run with the same added positional encodings is not yet available. We will include it in the revised Table 1(b).

AuthorsOpenReview ↗

Response to Reviewer 1xbU - Part 3 / 3

5. “How do its gains change with the training loss? Could you report the mAP gain from scoring for NTP-only training and Gaussian supervision with different kernel widths and mixing weights?” (Q3)

Confidence-scoring gains under different losses. We compare matched FastVLM2-4B Stage-2.0 checkpoints trained with NTP-only or our Gaussian supervision under the longer training schedule. The relevant comparison is the mAP gain from adding confidence scoring for each loss:

Training objectiveCOCO mAP, score-free ↑COCO mAP, with confidence ↑Gain from scoring ↑
NTP-only21.330.9+9.6
Gaussian DA-SoftCE (ours)22.832.8+10.0

Confidence scoring consistently improves COCO mAP under both training objectives, demonstrating that its effectiveness does not depend on Gaussian supervision.

We additionally compare the localization accuracy of the two training objectives at Stage 2.0:

Metric ↑NTP-onlyGaussian DA-SoftCE (ours)Δ
COCO F1 @ IoU 0.557.160.0+2.9
COCO mAP (with confidence)30.932.8+1.9
LVIS F1 @ IoU 0.552.655.2+2.6
LVIS mAP (with confidence)20.822.6+1.8
Avg. referring expressions68.369.4+1.1

Gaussian supervision further improves localization performance across COCO, LVIS, and referring expressions, while our training-free confidence estimator provides consistent gains regardless of the training objective.

Across training stages, we also measure confidence-scoring gains on checkpoints from Stage 2.0 through RL:

COCO mAP ↑Stage 2.0Stage 3.0Stage 4.0After RL
Score-free ranking22.826.831.236.0
Geometry-aware confidence (ours)32.836.040.244.5
Δ mAP (ours − score-free)+10.0+9.2+9.0+8.5

Across these checkpoints, scoring consistently improves COCO mAP by +8.5 to +10.0 points, including +8.5 after RL, despite changes in checkpoint quality.

Gaussian width × mixing weight. Using the same ablation-scale setup as Table 2, we measured RefCOCO average accuracy:

Gaussian kernel widthα = 0.3α = 0.5α = 0.7α = 0.9
167.171.175.473.5
371.172.674.970.7
5 (selected)72.874.875.976.4
1074.675.175.672.7

Width 5, α = 0.9 performs best (76.4), matching Table 2 and exceeding NTP-only (71.9) by +4.5 points. Widths 5 and 10 outperform NTP-only across all tested α values. The shorter ablation setup does not provide dense-grounding confidence mAP; repeating every hyperparameter setting under the full training schedule is computationally prohibitive, so we report RefCOCO accuracy for this sweep.

The authors' response addressed most of my concerns, so I have raised my score.

Comments on the paper

1 comment

Dear reviewers,

It seems that the authors have carefully responded to your reviews.

If you haven't already, please take your time to go over them and let authors know what you think, to your earliest convenience.

Best,
AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.