- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FastVLM2 extends FastVLM to support native-resolution inputs and efficient dense visual grounding. It adapts pretrained hierarchical vision encoders through lightweight positional modifications during VLM fine-tuning, without additional vision pretraining or image tiling. For grounding, it uses compact quantized location tokens, a Gaussian distance-aware loss to improve localization, and a training-free, geometry-aware score for ranking predicted boxes. The paper validates these components through encoder and loss ablations, VQA and grounding benchmarks, and latency measurements on an M1 Max MacBook. It also applies the box-scoring method to two independently trained grounding models, where it improves confidence-ranked COCO mAP.
Critical strengths
-
A strong efficiency–accuracy trade-off. FastVLM2-4B achieves nearly the same average VQA score as Qwen3-VL-4B (81.7 vs. 82.1) with substantially lower prefill latency on the tested device (1.2 vs. 5.6 seconds), while performing better on the reported grounding averages. This supports the paper’s practical case for a generalist on-device model.
-
A transferable box-confidence method. The training-free geometry-aware score improves COCO mAP by 8.3–8.6 points over score-free ranking across three independently trained models. Because it improves the ranking of unchanged predicted boxes without retraining, the result supports a contribution beyond FastVLM2’s particular model architecture.
Critical weaknesses
-
Limited methodological novelty. FastVLM2 retains the overall FastVLM architecture and combines existing positional mechanisms with quantized coordinates already used by Rex-Omni. Its grounding loss and confidence score are useful refinements, but the paper’s main contribution appears to be an effective system integration rather than a substantial new modeling principle.
-
Incomplete attribution of the final gains. The native-resolution ablations in Section 4.1 use Qwen2.5-1.5B, and the grounding-loss ablations in Section 4.2 use Qwen2.5-3B; the main results use Qwen3 decoders after SFT and RL. Without a matched baseline under the final training recipe, it is difficult to determine how much of the reported accuracy advantage comes from the proposed components rather than the changed model and training setup. This weakens the evidence for the paper’s central method claims.
Questions for authors
- Could the authors provide the key component ablations using the Qwen3 decoder employed in the final FastVLM2 models? The current ablations use Qwen2.5, so it is unclear whether the observed gains carry over to the Qwen3-based models reported in the main results.
2.Interpretation of latency: Appendix G measures decoding with a fixed seven-box budget, while the headline 7.3× comparison uses selected accuracy–resolution operating points. How representative is that speedup of end-to-end latency when models generate their natural, variable-length outputs on the grounding benchmark?
3 comments
Response to Reviewer NchQ - Part 2 / 2
3. “How representative is that speedup of end-to-end latency when models generate their natural, variable-length outputs on the grounding benchmark?” (Q2)
Appendix G's fixed-seven-box evaluation controls decoding length, while the reported 7.3× FastVLM2-1.8B versus Rex-Omni-3B comparison uses selected accuracy–resolution operating points. To assess realistic inference, we benchmarked FastVLM2-4B, Qwen3-VL-4B, and Rex-Omni-3B on the same 100 COCO validation images (Apple M1 Max, MLX; 4-bit LLM, FP16 vision encoder). Measurements include preprocessing, vision encoding, prefill, natural-length decoding, and postprocessing, including confidence scoring. This precision preserves FastVLM2 grounding accuracy (73.4 vs. 73.6 F1 @ IoU 0.5 with full FP16).
| Matched 100-image COCO subset | Qwen3-VL-4B | Rex-Omni-3B | FastVLM2-4B (ours) |
|---|---|---|---|
| Mean end-to-end latency (s) ↓ | 7.168 | 1.150 | 0.775 |
| Median end-to-end latency (s) ↓ | 1.997 | 1.073 | 0.632 |
| p90 / p99 latency (s) ↓ | 37.85 / 47.11 | 1.59 / 2.46 | 1.23 / 2.09 |
| Mean prefill (s) ↓ | 0.562 | 0.674 | 0.319 |
| Mean decoding (s) ↓ | 6.590 | 0.457 | 0.429 |
| Mean visual tokens ↓ | 271 | 357 | 153 |
| Mean output tokens ↓ | 604 | 53.1 | 44.5 |
| FastVLM2 improvement over | Mean end-to-end | Median end-to-end | Prefill |
|---|---|---|---|
| Qwen3-VL-4B | 9.25× faster | 3.16× faster | 1.76× faster |
| Rex-Omni-3B | 1.48× faster | 1.70× faster | 2.11× faster |
Qwen3-VL generates much longer outputs (604 vs. 44.5 mean tokens), with 10/100 examples reaching the generation cap; we therefore report median and tail latencies rather than attributing the full mean speedup to vision encoding. Rex-Omni is the more informative token-format comparison: it also uses compact locations and generates similar-length outputs (53.1 vs. 44.5 tokens), yet FastVLM2 achieves 2.11× faster prefill and 1.48× faster mean end-to-end inference, despite its larger decoder (4B vs. 3B).
Confidence estimation adds approximately 0.5 ms total for seven boxes (28 coordinates), already included in end-to-end measurements. FastVLM2 also uses 29.4 J/image and 3.47 GB peak MLX memory, versus 241.4 J / 5.42 GB for Qwen3-VL and 44.6 J / 4.04 GB for Rex-Omni.
These results show that FastVLM2's efficiency advantage persists with natural-length grounding outputs, while the Qwen3 ablation supports the proposed loss. We will incorporate these measurements and distinguish the fixed-budget, selected-operating-point, and natural-output evaluations in the revised paper.
Maintain Weak Acceptance
Thank you for the clarifications and additional experiments. The Qwen3-4B ablation and natural-length end-to-end latency measurements substantially address my main empirical concerns. The comparison with Rex-Omni is particularly informative, showing an efficiency advantage with similar output lengths. Please clearly distinguish these results from the original 7.3× comparison in the revised paper.
While my reservation about methodological novelty remains, I recognize the paper’s practical value and maintain my rating of 3 (Weak acceptance).
Response to Reviewer NchQ - Part 1 / 2
We thank the reviewer for recognizing FastVLM2's accuracy–latency trade-off and transferable confidence estimator. Below, we clarify our methodological contributions and present new Qwen3-based ablations and end-to-end on-device measurements with natural-length grounding outputs.
1. “Limited methodological novelty.” (W1)
We build on FastVLM and do not claim CoordConv, 2D RoPE, or quantized coordinates as individually new. Our contributions address three complementary challenges in efficient generalist VLMs:
Native-resolution hierarchical encoders. To the best of our knowledge, FastVLM2 provides the first fine-tuning-only recipe for adapting pretrained hierarchical convolutional (ConvNeXt) and hybrid (FastViTHD) encoders to variable native resolutions and aspect ratios within a single checkpoint, without additional vision pretraining, tiling, or explicit layout tokens. Unlike fixed-resolution FastVLM checkpoints, this enables multiple accuracy–latency operating points from the same model. The recipe combines lightweight positional adaptations with a staged training curriculum.
Geometry-aware supervision for compact location tokens. Gaussian distance-aware supervision substantially improves localization accuracy when training with quantized location tokens, mitigating the accuracy loss from quantization while preserving the decoding efficiency of compact outputs (over 60% fewer tokens).
Transferable, training-free box confidence. Our geometry-aware estimator uses existing location-token distributions to score predicted boxes. It requires no additional training, confidence head, or forward passes and generalizes to other models using compact location tokens, improving COCO mAP by over 8 points across three independently trained models.
2. “Could the authors provide the key component ablations using the Qwen3 decoder employed in the final FastVLM2 models?” (Q1) “Incomplete attribution of the final gains.” (W2)
We performed a controlled Qwen3-4B grounding-loss ablation using the final recipe's full Stage 1.0/1.5/2.0 schedules (5k/38k/33k steps), rather than the shorter Qwen2.5 ablations in the submission. The runs differ only in their grounding objective: NTP-only versus Gaussian distance-aware supervision.
Gaussian supervision improves both dense grounding and referring expressions with the Qwen3-4B decoder, supporting the transfer of this contribution to the final decoder family and longer training schedule. This experiment isolates the loss contribution through Stage 2.0.