acceptodds
Under review as a conference paper at ICLR 2027

RVLM: Fewer Tokens, Better Resolution Scaling, Fine-Grained Spatial Grounding

Abstract

Vision-Language Models (VLMs) tokenize images using fixed spatial grids, so visual token count grows with pixel area rather than scene content. We propose to group neighboring patches based on their visual appearance, allowing tokens to represent coherent image regions rather than fixed spatial windows. We introduce RVLM, a VLM trained to use regions rather than patches as its visual units. RVLM partitions images using Felzenszwalb-Huttenlocher segmentation, assigns each ViT patch to a region, and pools patches within each region into a variable-length sequence of visual tokens. The patch-to-region assignment is retained to support spatially precise prediction through coarse-to-fine decoding. Across tasks spanning VQA, pointing, counting, segmentation, and depth estimation, RVLM matches or improves example-weighted task performance while using 21.5% fewer visual tokens than an identically trained patch baseline. The advantage grows with image resolution, reaching a 5.3-point gain while using 51.8% fewer visual tokens at the highest-resolution setting. The gains are largest on spatially demanding tasks, where prior token-reduction methods degrade most sharply. RVLM adds no model parameters and improves inference throughput by up to 35.9%. These results establish regions as a scalable alternative to patch-based visual representations while preserving the spatial precision required for grounded prediction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language models, region-based representations

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_6d9xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RVLM, a vision-language model that replaces the fixed spatial pooling of ViT patch features with content-adaptive region pooling. The method first applies a training-free Felzenszwalb-Huttenlocher (FH) image segmentation algorithm, assigns each ViT patch to a spatially connected region, and pools patch features within each region into variable-length region tokens.

Critical strengths
  1. The paper evaluates a broad range of tasks, including image QA, document understanding, pointing, counting, segmentation, and depth estimation. The evaluation goes beyond standard VQA benchmarks and covers settings where spatial information is critical.
  2. The paper includes detailed analyses of FH segmentation. These experiments provide useful evidence regarding the practical trade-offs of the proposed design.
Critical weaknesses
  1. The novelty appears limited relative to prior region-based or content-adaptive tokenization methods. The main distinction is the use of a training-free FH segmentation algorithm as a CPU-side preprocessing step to obtain complete, spatially connected regions before the VLM forward pass. However, this seems to be primarily a design choice rather than a fundamentally new region-tokenization formulation. Furthermore, CPU-side preprocessing is not inherently advantageous.
  2. RVLM still relies on a standard patch-based ViT to encode all image patches, and region tokens are formed only by pooling the resulting patch features. Thus, the method is better viewed as post-encoder patch-feature aggregation rather than a truly region-native visual representation.
Questions for authors
  1. In the segmentation setting, how are patch-level and 4×4 cell-level logits produced after patches have been pooled into region tokens? Does the segmentation head access the original ViT patch embeddings? Additionally, where do the cell features come from?

  2. The paper does not clearly describe the task-specific model architectures (e.g., task-specific heads), task sampling, loss weights, or how objectives across QA, OCR, pointing, counting, segmentation, and depth are balanced. Could you explicitly describe these details.

  3. The claimed grounding results are coupled with multi-task co-training involving tasks such as segmentation, pointing, and counting. It is therefore unclear whether region tokenization itself improves visual grounding (e.g., on RefCOCO) in more standard VLM settings.

2 comments

Thanks for your detailed reply.

  1. For the novelty, my concern is what new insight introduced in this paper. However, I do not consider the response "making it work" to technical novelty, although it can be regarded as a contribution.
  2. For the CPU-side preprocessing, My point is simply that CPU-side is not inherently an advantage. The comparison with T-REN dose not address this concern. FN could also be optimized and implemented on the GPU. CPU implementation is therefore be presented as an engineering choice.
  3. For visual grounding, to clarify my intended scope, my question concerned detection performance in standard VLM settings, not the fairness of comparison.
  4. Thank you for the addition details of the experiments. The manuscript I reviewed still need a major revision to address these concerns.

Thank you for engaging with our responses and providing further clarification.

  1. Key insight: For an effective general-purpose region-based VLM, regions must cover the full image, stay spatially connected as the model learns, and preserve fine structure such as small text, while semantic grouping adds little. Evidence in Sec. 4.4, Table 4.

    Practical consequence: A fast, training-free pixel segmenter suffices these requirements, which lets RVLM use fewer tokens and scale to higher resolutions while matching or improving grounding.

    The revised abstract and contributions now lead with this insight.


  1. We agree that running FH on the CPU is an engineering choice, and the revised paper now describes it as such (Sec. 2). It is nonetheless a practically important one, because it keeps RVLM cheap to deploy. On the CPU, region construction uses no accelerator time, which matters in real-world settings where GPU time is the main cost. CPU dataloader workers hide FH behind model execution (Sec. 4.2), so RVLM’s end-to-end throughput gains (up to 35.9%; Table 3) already include region construction. A GPU implementation would instead compete with the model for accelerator time.

  1. We do not claim that region tokens improve grounding when coordinates are generated as text, and our 1.7B runs confirm they do not. Regions trail patches by 4.8 points on PixMo-Point (72.7 vs. 77.5; App. B.4, Table 12), because the LLM must infer locations from pooled region embeddings that can span up to 16 patches (Sec. 3.1, Table 9). However, with RVLM’s coarse-to-fine decoding, regions instead surpass patches on PixMo-Point (86.6 vs. 84.4 at 1.7B; 86.4 vs. 85.2 at 4B) and match or beat them in every grounding family (Table 2).

    We also evaluate referring expression comprehension on the RefCOCO, RefCOCO+, and RefCOCOg val splits ([email protected]).

    ModelRefCOCORefCOCO+RefCOCOg
    PVLM95.3292.8392.99
    RVLM94.9592.7393.24

  1. We have uploaded a revised manuscript addressing all raised concerns (changes highlighted).

    We hope the revised manuscript addresses your concerns, and would be grateful if you would take it into account in your final assessment.

If any concern remains, please let us know which, and we will address it before the discussion closes.

Reviewer_GgSNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents the RVLM method that aims to better represent the visual tokens. It uses the training-free Felzenszwalb-Huttenlocher segmentation method to get the list of regions, where each region is content coherent and associated with corresponding patches used for ViT. The RVLM supports patch-to-region assignment such that the downstream spatial grounding tasks are fully supported. The method is intuitively reasonable and takes advantage of scaling over image resolution, since the regions are less dependent on the image size than the patches. Experiments show that RVLM achieved 21.5% visual token reduction on average compared to the traditional patch-based VLMs.

Critical strengths
  • The proposed RVLM intuitively makes sense to achieve the highlighted advantages, such as the better scaling, computational efficiency, and training-free dependency.

  • The model is benchmarked on multiple datasets, including reasoning, understanding, grounding, and 3D vision datasets. All showed the best performance. Besides, the throughput is also comprehensively evaluated and shows superiority to existing approaches such as FastV, ToMe, and VisionZip.

Critical weaknesses
  • Technical novelty is limited. The proposed method is more like data preprocessing by using the traditional FH tool to get segmentation masks for region-based patch token pooling. This naturally raises the concerns of the generality in practice for various scenarios that need VLM models. For example, is the FH applicable to satellite images, pathological slices, surveillance videos, etc? Apart from introducing FH to do attention pooling, there are no other technical contributions presented in this paper.

  • Following-up the previous concerns, this paper can be strengthened to think deeply over these aspects: 1) can the FH method be adapted as a learnable module such that the entire pipeline is learned end-to-end? This will further improve the generalization capabilities of the model. 2) what if the input are hours-long videos? In this case, will the benefits of using FH segmentation still hold? 3) it’s good to see that position encoding is considered, however, what if the VLM backbone does not support M-RoPE? Looks like this paper limits the scope to the Qwen series VLMs.

  • It’s better to validate the method across different model architectures (dense and sparse), model sizes, and modalities (text-only, image-text, video-text, etc).

  • Since FH segmentation is the introduced module, the experimental sections should discuss or analyze it a bit. For example, what are the practical heuristics when using FH? What are the time or memory costs of this tool, especially when the image contains dense details such as high-resolution satellite images?

Questions for authors

N/A

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and responses

We thank both reviewers for their feedback. We have uploaded a revised manuscript in which all changes are color-coded: red for Reviewer 6d9x, blue for Reviewer GgSN, and purple for changes addressing both. Below we summarize what the paper contributes, what changed, and how each concern was addressed.

What the paper contributes

  • Insights into what makes region tokens work for general-purpose VLMs. Analyzing ten region construction strategies, we find that regions must cover the full image, stay spatially connected during end-to-end training, and preserve fine structure such as small text, while semantic grouping adds little. Segmenters that leave pixels uncovered cannot support pointing at those locations, and regions derived from the VLM's own features fragment as the encoder trains. In contrast, learned segmenters and clustering on a separate frozen encoder do not outperform classical FH segmentation, which is fast, adds no model parameters, and preserves small text better than SLIC.

  • An efficient region-based VLM. Against an identically trained patch baseline across 35 tasks, RVLM uses 21.5% fewer visual tokens while improving the average score (+0.6) and inference throughput (up to +35.9%). Its advantage grows with resolution, reaching +5.3 points with 51.8% fewer tokens at the highest resolution. It also outperforms pruning and merging methods at matched token budgets on pointing, counting, segmentation, and depth (Fig. 3).

Key changes to the manuscript

  • Key insight foregrounded. The abstract, contributions, Sec. 4.4 (new summary paragraph), and conclusion now state the finding above explicitly.
  • Technical design made explicit (Sec. 3.1, App. B.4). Naively replacing patch tokens with region tokens collapses coarse-to-fine pointing to 10.0 on PixMo-Point, compared with 84.4 for patches and 86.6 with our design. Sec. 3.1 now explains why region splitting and exhaustive, exclusive patch assignment are required.
  • Grounding heads detailed (Sec. 3.1, App. A.3). The grounding heads read the unpooled ViT patch features, which are retained after pooling. App. A.3 specifies how region, patch, and cell logits are computed.
  • New experiments.
    • Unseen domains (App. B.5). On pathology and satellite images, RVLM is on par with the patch baseline using about half the tokens. FH also achieves higher boundary recall than SAM and covers every pixel.
    • Alternative position encoding (App. B.6). It performs within 0.3 points of M-RoPE.
    • Model scale (App. B.7). At 1.7B, RVLM improves pointing by 2.2 points.
    • Text-only and video (App. B.8). Text-only performance is essentially unchanged, and on video, without video training, RVLM is on par with the patch baseline while using 3× fewer tokens.
  • Clarifications.
    • CPU execution of FH is now described as an engineering choice (Sec. 2).
    • FH practical heuristics are added (App. B.1).
    • The LLM’s 1D RoPE extension to M-RoPE is documented.
    • Limitations stated explicitly (App. C).

Where each concern is addressed in the revised manuscript (all points are also addressed in our OpenReview responses)

  • Reviewer GgSN.

    • Technical novelty → App. B.4 and Sec. 4.4.
    • Generality → unseen domains (App. B.5) and Table 7.
    • Learnable FH → learned partitions fragment (Sec. 4.4, Table 4).
    • M-RoPE and Qwen dependence → App. A.1, B.6.
    • Sizes, modalities, long video → App. B.7, B.8, conclusion.
    • FH heuristics and cost → App. B.1, Sec. 4.2.
  • Reviewer 6d9x.

    • Novelty → key insight and design ablation (Sec. 3.1, 4.4, App. B.4).
    • CPU preprocessing → engineering choice (Sec. 2).
    • Segmentation logits → App. A.3.
    • Detection in standard settings → text-coordinate result and limitation (App. B.4, C)

Why we believe the paper merits acceptance

  • It offers a transferable finding, backed by a controlled comparison of ten strategies, about what region tokenizers for trainable VLMs should optimize: coverage, connectivity, and fine structure rather than semantic grouping.
  • Its evaluation is broad and controlled. Across 35 tasks with an identically trained baseline, region tokens reduce visual tokens without loss and scale better with resolution.
  • It preserves spatial precision where existing token reduction methods fail, generalizes across domains, scales, and modalities, and adds no model parameters, making it easy to adopt.

We are grateful for the reviewers’ engagement and hope the revised manuscript addresses their concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.