acceptodds
Withdrawn from ICLR 2027

Beyond Semantic Alignment: Towards Dense Instance Perception in Vision-Language Models for Open-World Counting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language Models (VLMs) achieve strong semantic understanding through large-scale image-text alignment, yet their ability to distinguish and enumerate nearby visual instances degrades substantially as object density increases. We investigate this semantic-to-instance perception gap through a unified language-guided counting benchmark built from four existing counting datasets and a controlled evaluation of representative visual encoders. Our analysis reveals that stronger semantic representations benefit counting mainly in sparse scenes, while their advantages diminish under increasing crowding. To alleviate this limitation, we propose DIP-CountVLM, a Dense Instance Perception Counting VLM for open-world scenarios. DIP-CountVLM combines Instance-Aware Feature Fusion (IAFF), which adaptively aggregates hierarchical visual features to recover instance-sensitive cues, with Dense Instance Token Reconstruction (DITR), which leverages image guidance to reconstruct spatially denser visual tokens. Extensive experiments show that DIP-CountVLM improves overall counting performance over a SigLIP2-based VLM baseline, with the most consistent gains in low- and medium-density regimes. Further analyses demonstrate that hierarchical feature fusion is most effective when coupled with dense token reconstruction, while extremely dense scenes remain challenging. These findings provide a systematic analysis of the visual representation bottlenecks underlying language-guided counting and a practical approach for alleviating them.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision & Multi-modal

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vekHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DIP-CountVLM for open-world counting. It introduces IAFF to fuse multi-layer features and DITR to reconstruct dense visual tokens through input-representation cross-attention, aiming to move beyond global semantic alignment toward instance-level perception. It also builds DenseCountBench by reorganizing four counting datasets into low-, medium-, and high-density splits. Experiments report improved overall HitRate.

Critical strengths
  1. The paper is well-organized and easy to follow and understand.
  2. The motivation that global image-text alignment in VLMs is insufficient for dense instance counting is reasonable.
  3. The method direction is intuitive. The authors exploit complementary multi-layer features and attempt to recover spatial details.
  4. The experiments are relatively systematic, covering component analysis, dynamic/fixed fusion, frozen/trained DITR, and feature-layer selection on various scenes.
Critical weaknesses
  1. The novelty is relatively limited. For instance, IAFF is essentially a simple weighted fusion of multi-layer visual features, which is similar to popular multi-level feature methods (e.g., FPN [Lin et al., CVPR 2017], and MMFuser [Cao et al., ICPR 2026]), while DITR performs image-guided input-representation cross-attention, which is similar to recent feature upsampling methods (e.g., JAFAR [Couairon et al., NeurIPS 2025] and AnyUp [Wimmer et al., ICLR 2026]). The authors are recommended to provide a discussion or comparison.
  2. The central dense-instance claim is not convincingly supported by experiments. The gains of DIP-CountVLM in tab. 1 are concentrated mainly in the low-density regime, while medium-density results are inconsistent, and high-density results show essentially no meaningful improvement. A similar issue arises in the ablation study in tab. 2. In addition, the main paper figures also do not directly visualize whether DIP-CountVLM produces better instance-level representations/attention than the compared VLMs (fig. 1 and fig. 3 mainly provide comparisons among general VLMs).
  3. Some descriptions of the main components are not clear enough to understand in the main paper. For instance, DITR involves a feature-agnostic layer, downsampling, and Q/K/V branches, but the paper does not formally define these operations. Some important architectural choices are not explicitly specified in the method section, for instance, how the feature-agnostic layer is implemented and the downsample/upsample ratios.
Questions for authors

In addition to the weaknesses above, the reviewers are also willing to discuss the following issues with the authors:

  1. Is HitRate@100% too strict as the sole evaluation metric? Since this metric requires exact count matching, many methods achieve near-zero scores, especially in dense scenes, which limits its ability to differentiate methods. Do the authors consider reporting more tolerant HitRate values or standard counting metrics such as MAE/MSE?
  2. How does DenseCountBench actually reflect the 'open-world' setting? Beyond repackaging existing counting datasets into language-guided queries, does the benchmark include synonyms, unseen categories, or natural language descriptions of category attributes? Without this, it is unclear whether the benchmark truly evaluates open-world generalization rather than category-specific counting.
Reviewer_3zqeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the limitations of VLMs in dense, language-guided open-world object counting. It proposes DIP-CountVLM, which combines IAFF and DITR to enhance instance-level perception. Experiments on four counting benchmarks show improvements over a general-purpose SigLIP2-based VLM baseline, primarily in low- and medium-density scenarios, with limited gains in highly dense scenes.

Critical strengths

None

Critical weaknesses
  1. The evaluation is unfair. The comparison focuses almost exclusively on general-purpose VLMs while omitting comparisons with existing text-guided and open-world counting methods, such as CLIP-Count [1] and CountGD [2]. Since these methods address essentially the same task, their exclusion makes it difficult to assess the competitiveness of the proposed method and distinguish it from existing approaches. Although controlled comparisons among VLM backbones are useful, they are insufficient to establish the effectiveness of a new method.
  2. The experimental results do not support the paper's claim of improving dense instance perception. On high-density subsets (see Tab. 2), DIP-CountVLM achieves almost no improvement over the baseline: 0.1% vs. 0.1% on IOCFish5K, 1.4% vs. 1.4% on UnderwaterFish, and 0.3% vs. 0.2% on UNICBench. The improvement on DeepFish comes from a dataset consisting almost entirely of low-density samples. In addition, the ablation studies provide limited evidence for the effectiveness of IAFF and DITR. IAFF alone degrades performance on 3/4 benchmarks, while DITR produces inconsistent improvements. Moreover, the full model does not outperform the baseline consistently.

[1]. https://arxiv.org/pdf/2305.07304 [2]. https://arxiv.org/pdf/2407.04619

Questions for authors
  1. Why are all the numbers for DIP-CountVLM bolded in Tab. 2, even though they are not the best in each column?
Reviewer_YPHTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper compares seven frozen visual encoders paired with two language backbones for language-guided counting. All use the same instruction-tuning protocol and are tested on four existing datasets. DIP-CountVLM adds adaptive fusion of intermediate SigLIP2 features and image-guided reconstruction of a denser token grid. It improves exact-count accuracy over Qwen3-8B/SigLIP2 on all four datasets, most clearly on DeepFish. The authors use these results to argue that semantic alignment alone leaves a gap in dense instance perception.

Critical strengths
  1. The encoder comparison in Table 1 is useful. Keeping the language backbone and instruction-tuning protocol fixed gives a clearer picture of the visual encoder than comparisons between unrelated VLMs. Repeating the analysis with Qwen3-8B and Llama2-7B adds weight to the observations. The poor exact-count performance above 50 objects is worth documenting, although it is a result about the complete counting system.
  2. The DeepFish improvement from 49.1% to 56.7% is substantial. Table 2 also makes it possible to judge the two components separately: hierarchical fusion alone does not reliably help, and its combination with reconstruction works better on some datasets. The abstract gives a reasonable account of where the gains occur, acknowledging that they are most consistent in low- and medium-count scenes.
Critical weaknesses
  1. My main concern is that the evaluation does not isolate dense instance perception. Section 4.1 defines density by object count and reports only exact-count accuracy. A prediction off by one receives the same score as a much larger error. Exact-match accuracy can therefore fall as counts increase even if relative counting error stays unchanged. Object size, spacing and overlap are not controlled either. The high-count results give little evidence that the proposed method resolves this problem: accuracy stays at 0.1% on IOCfish5k and 1.4% on UnderwaterFish, and changes from 0.2% to 0.3% on UNICBench (Table 1). DeepFish, where the gain is largest, contains only six medium-count images and no high-count images (Figure 5). I can see evidence of better counting on some distributions, but cannot tell whether it comes from separating crowded instances or predicting small counts better. The proposed explanation needs relative-error or instance-level evidence, with resolution and token budget controlled.
  2. DITR is also difficult to assess from the description provided. Section 3.3 and Appendix A.3 define it as D(I,F), without specifying the reconstruction equations, window size, feature dimensions, upsampling factor or final token count. Figure 2 does not supply these details. The training account needs clarification too: Section 4.1 describes general LLaVA instruction data, while the ablations refer to an undefined counting objective. Several results then conflict with the accompanying interpretation. Identical DITR-only UnderwaterFish scores of 10.1/3.3/0.0 produce an overall score of 2.8 in Table 2 and 3.8 in Table 4. Dynamic fusion underperforms fixed fusion on UnderwaterFish, 3.8 versus 4.2 (Table 3), and the full model underperforms DITR alone on UNICBench, 11.5 versus 11.8 (Table 2). These differences matter when the paper argues that both components are needed. The method specification and ablation account need to be corrected before that argument can be assessed.
Questions for authors
  1. Please give the complete DITR computation, including input and output grid sizes, window attention and token count. Is it based on an existing image-guided feature-upsampling method?
  2. What is the counting objective mentioned in Tables 4–5? Which training examples and supervision were used for each variant? Were benchmark labels used for training or model selection?
  3. Is the DITR-only UnderwaterFish score 2.8 or 3.8? Please reconcile Tables 2 and 4 and explain which ablation conclusions hold after correction.
  4. Using the existing predictions, do the improvements hold under relative-error tolerances or normalized absolute error within each count bin? Does error decrease above 50 objects despite the nearly unchanged exact-match scores?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.