- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents RVLM, a vision-language model that replaces the fixed spatial pooling of ViT patch features with content-adaptive region pooling. The method first applies a training-free Felzenszwalb-Huttenlocher (FH) image segmentation algorithm, assigns each ViT patch to a spatially connected region, and pools patch features within each region into variable-length region tokens.
Critical strengths
- The paper evaluates a broad range of tasks, including image QA, document understanding, pointing, counting, segmentation, and depth estimation. The evaluation goes beyond standard VQA benchmarks and covers settings where spatial information is critical.
- The paper includes detailed analyses of FH segmentation. These experiments provide useful evidence regarding the practical trade-offs of the proposed design.
Critical weaknesses
- The novelty appears limited relative to prior region-based or content-adaptive tokenization methods. The main distinction is the use of a training-free FH segmentation algorithm as a CPU-side preprocessing step to obtain complete, spatially connected regions before the VLM forward pass. However, this seems to be primarily a design choice rather than a fundamentally new region-tokenization formulation. Furthermore, CPU-side preprocessing is not inherently advantageous.
- RVLM still relies on a standard patch-based ViT to encode all image patches, and region tokens are formed only by pooling the resulting patch features. Thus, the method is better viewed as post-encoder patch-feature aggregation rather than a truly region-native visual representation.
Questions for authors
-
In the segmentation setting, how are patch-level and 4×4 cell-level logits produced after patches have been pooled into region tokens? Does the segmentation head access the original ViT patch embeddings? Additionally, where do the cell features come from?
-
The paper does not clearly describe the task-specific model architectures (e.g., task-specific heads), task sampling, loss weights, or how objectives across QA, OCR, pointing, counting, segmentation, and depth are balanced. Could you explicitly describe these details.
-
The claimed grounding results are coupled with multi-task co-training involving tasks such as segmentation, pointing, and counting. It is therefore unclear whether region tokenization itself improves visual grounding (e.g., on RefCOCO) in more standard VLM settings.
2 comments
Thank you for engaging with our responses and providing further clarification.
-
Key insight: For an effective general-purpose region-based VLM, regions must cover the full image, stay spatially connected as the model learns, and preserve fine structure such as small text, while semantic grouping adds little. Evidence in Sec. 4.4, Table 4.
Practical consequence: A fast, training-free pixel segmenter suffices these requirements, which lets RVLM use fewer tokens and scale to higher resolutions while matching or improving grounding.
The revised abstract and contributions now lead with this insight.
- We agree that running FH on the CPU is an engineering choice, and the revised paper now describes it as such (Sec. 2). It is nonetheless a practically important one, because it keeps RVLM cheap to deploy. On the CPU, region construction uses no accelerator time, which matters in real-world settings where GPU time is the main cost. CPU dataloader workers hide FH behind model execution (Sec. 4.2), so RVLM’s end-to-end throughput gains (up to 35.9%; Table 3) already include region construction. A GPU implementation would instead compete with the model for accelerator time.
-
We do not claim that region tokens improve grounding when coordinates are generated as text, and our 1.7B runs confirm they do not. Regions trail patches by 4.8 points on PixMo-Point (72.7 vs. 77.5; App. B.4, Table 12), because the LLM must infer locations from pooled region embeddings that can span up to 16 patches (Sec. 3.1, Table 9). However, with RVLM’s coarse-to-fine decoding, regions instead surpass patches on PixMo-Point (86.6 vs. 84.4 at 1.7B; 86.4 vs. 85.2 at 4B) and match or beat them in every grounding family (Table 2).
We also evaluate referring expression comprehension on the RefCOCO, RefCOCO+, and RefCOCOg val splits ([email protected]).
Model RefCOCO RefCOCO+ RefCOCOg PVLM 95.32 92.83 92.99 RVLM 94.95 92.73 93.24
-
We have uploaded a revised manuscript addressing all raised concerns (changes highlighted).
We hope the revised manuscript addresses your concerns, and would be grateful if you would take it into account in your final assessment.
If any concern remains, please let us know which, and we will address it before the discussion closes.
Thanks for your detailed reply.