acceptodds
Under review as a conference paper at ICLR 2027

CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes

Abstract

Do Vision-Language Models (VLMs) count what they see, or what they know about the world? We introduce CounterCount, a diagnostic dataset of paired factual and counterfactual (CF) images, in which the CF image contains an atypical count of a subject's attribute (e.g., a rabbit with four ears), together with annotations localizing the counted attributes. Humans answer both types of images nearly perfectly (98.9% and 97.5%), but across 16 VLMs (14 open-weight and 2 closed-source), average accuracy drops from 91.6% on factual to 49.5% on CF images, with most errors stemming from responses that match the canonical count, therefore revealing a bias towards knowledge priors rather than reliance on visual information. Based on these findings, we perform an analysis that reveals two interesting insights behind this gap. First, models exhibit a "System-1"-like response behavior, falling back on canonical counts after a shallow recognition of the subject. Second, model activations partially contain the visual evidence necessary to answer correctly, yet models underutilize it, attending less to the counted attributes when they provide biased answers. Input manipulations that break visual context, introduce cueing or reasoning, or direct amplification of the model's internal attention to relevant regions can reduce such biases. CounterCount goes beyond showing that VLMs fall back on knowledge priors, revealing that once a familiar context is recognized, models favor prior knowledge over available visual evidence rather than failing to encode that evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Language Models, Visual Attribute Counting, VLM Bias

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CHaTOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CounterCount, a diagnostic benchmark designed to investigate whether vision-language models (VLMs) rely on visual evidence or knowledge priors when counting object attributes. To reduce the confounding effects of general counting difficulty and visual ambiguity in existing counterfactual counting benchmarks, the authors construct 554 manually verified factual–counterfactual (CF) image pairs, covering 379 subjects, 228 attribute types, and 17 semantic categories. The dataset also provides region-level annotations, including segmentation masks and bounding boxes. The authors evaluate 16 VLMs alongside human participants. While the models achieve an average accuracy of 91.6% on factual images, their accuracy drops to 49.5% on CF images, with most incorrect responses matching the canonical attribute count. The paper further investigates this behavior through visual context blurring, instructional cueing, thinking mode, attention analysis, and attention modulation. Based on these experiments, the authors suggest that models tend to fall back on knowledge priors after recognizing familiar objects and may underutilize relevant visual evidence.

Critical strengths
  1. CounterCount provides a well-motivated and valuable diagnostic benchmark for studying the reliance of VLMs on visual evidence versus knowledge priors. When evaluating knowledge-prior bias through attribute counting, model errors may arise from either reliance on prior knowledge or limitations in visual counting ability, making it difficult to distinguish between these factors. CounterCount addresses this challenge by focusing on relatively small attribute counts, constructing controlled factual–CF image pairs, and manually verifying the visibility and correctness of the edited attributes. These design choices help reduce the influence of general counting difficulty and visual ambiguity, enabling a more targeted assessment of how VLMs respond when visual evidence conflicts with canonical knowledge. Furthermore, the dataset covers a diverse range of subjects and attributes and provides localized annotations for fine-grained analysis. These features make CounterCount a useful resource for the community to further investigate visual grounding and knowledge-prior bias in VLMs.
Critical weaknesses
  1. The distinction between knowledge-prior bias and general visual counting difficulty could be better substantiated. A central motivation of CounterCount is to reduce the confounding effects of general counting difficulty by focusing on relatively small attribute counts. While this is a reasonable design choice, the paper's results suggest that counting-related factors may still influence the observed performance differences. For example, Figure 6 shows that CF accuracy decreases as the attribute count increases, while attribute addition and removal exhibit substantially different bias rates. The authors already provide useful analyses of these effects, but the results primarily characterize performance differences across conditions rather than disentangle the contributions of counting difficulty and knowledge-prior reliance. A more controlled analysis, for example by comparing bias under matched attribute counts and editing conditions, would help clarify the extent to which the observed failures specifically reflect knowledge-prior bias and further strengthen the benchmark's diagnostic validity.

  2. The practical diagnostic advantages over existing benchmarks could be demonstrated more directly.The paper provides a clear comparison with related benchmarks in the Related Work section and Appendix A.2, highlighting CounterCount's broader subject and attribute coverage, controlled image pairs, and localized annotations. These are meaningful design advantages, and the attention-based experiments demonstrate the usefulness of the additional annotations. However, the comparisons primarily focus on dataset characteristics, while the extent to which these advantages translate into improved diagnostic capabilities remains less clear. Related benchmarks, such as Vo et al. (2025b) and CDH-Bench (Chen et al., 2026), also investigate conflicts between visual evidence and prior knowledge, including counterfactual attribute counting. A direct comparison under comparable counting tasks and evaluation settings would help establish whether CounterCount provides additional insights into model behavior or more effectively differentiates models in terms of their reliance on visual evidence and knowledge priors. Such evidence would further strengthen the case for CounterCount as a valuable benchmark for future research.

Questions for authors
  1. Distinguishing knowledge-prior bias from counting difficulty. Figure 6 reports the effects of attribute count and editing direction on CF accuracy and canonical-count bias. Since these factors may be correlated, could the authors provide an additional analysis under matched counting and editing conditions? In particular, does the observed knowledge-prior bias remain consistent after controlling for these factors?

  2. Incremental diagnostic value over existing benchmarks. The paper clearly demonstrates the design advantages of CounterCount in terms of dataset coverage and localized annotations. Could the authors further clarify its practical diagnostic advantages over existing counterfactual counting benchmarks? For example, would a comparison using the same VLMs and comparable counting tasks reveal additional model behaviors or distinctions that are less apparent in existing benchmarks?

  3. Generalizability of the findings. All CounterCount images are generated using Nano-Banana, which facilitates controlled factual–CF construction. However, different image generation or editing methods may produce different visual characteristics. Could the authors discuss how these differences might affect the observed knowledge-prior bias and the extent to which the findings are expected to generalize to counterfactual images from other sources?

4 comments

1. Prior bias under matched counts and edits

We thank the reviewer for their feedback and for recognizing the value of CounterCount. Figure 6a already compares factual and CF accuracy at the same count. The number of attributes to count is identical, yet factual accuracy is higher than CF accuracy at every count, showing that the high factual accuracy reflects prior knowledge rather than actual counting. As for editing conditions, we understand that the reviewer refers to the addition and removal of attributes. Following the reviewer’s suggestion, we further analyzed bias under the same count for the two modifications, as shown in the table below:

CF countAccuracy (addition)Accuracy (removal)Bias (addition)Bias (removal)
0—45.5%—46.6%
1—24.7%—69.9%
271.9%34.6%23.6%61.1%
356.8%11.9%28.6%76.2%
4+37.2%16.4%44.2%76.6%
Overall55.1%29.2%32.2%64.2%

The table shows that the bias remains after controlling for both factors. It is present at every count for both edits, and for the same count it is much higher for removal than for addition (e.g., 61.1% vs 23.6% at a count of two). Counting difficulty cannot explain this difference, for three reasons:

  • Same count, different outcome. At a fixed CF count, the number of attributes the model has to count is identical for both edits. If counting were the limiting factor, accuracy would be similar. Instead, it is far lower for removal: 34.6% vs 71.9% at a count of two, and 11.9% vs 56.8% at a count of three.
  • The smallest counts are not the easiest. Counts of zero and one should be easiest to count, yet accuracy is only 45.5% and 24.7% because it is removal.
  • The errors go to the prior. In the removal cases, most answers return the canonical count (61.1% to 76.6% for counts of two and above). What differs between the two edits is therefore not the number of attributes to count, but how the image relates to the prior, as models fill in a missing attribute from prior knowledge more often than they overlook an added one. We added this analysis to Appendix C.9.

2. Diagnostic advantages over existing datasets

We thank the reviewer for recognizing that the region-level annotations and the attention-based experiments they enable are useful. Below, we state concretely what the existing Counterfactual counting datasets , VLMBias (Vo et al., ICLR 2026; cited as Vo et al., 2025b in our submission) and CDH-Bench (Chen et al., 2026), do not allow, and what CounterCount adds:

  • Limited subjects and attributes. VLMBias has 332 unique counting images covering about 19 attributes in 7 categories. CDH-Bench has 3 categories, of which only one is counting, with 100 counting pairs covering 66 subjects and 60 attributes. CounterCount has 554 pairs covering 379 subjects and 228 attributes in 17 categories.

  • No fine-grained annotations for diagnosis and mitigation. VLMBias and CDH-Bench do not localize the counted attribute, so they can only evaluate models or test simple mitigations (prompt instructions and background removal in VLMBias). Our verified subject mask and attribute bounding box for every CF image allowed us to measure how much attention models give to the counted region, showing that they under-attend to it when they fail (Section 4.2), and to test two region-targeted mitigation strategies: blurring the visual context while keeping the attribute intact (Section 4.1.2) and modulating attention toward the counted region (Section 4.2). This helps analyze and understand the behavior of VLMs more precisely.

  • Human Evaluation CDH-Bench does not provide any human evaluation, and VLMBias reports a human study only on animals category. CounterCount provideshuman evaluation to have a solid upper-bound on how far we can push the models.

  • No controlled factual–CF pairs. VLMBias evaluates unmodified images only as a 66-image sanity check, and CDH-Bench pairs are generated separately so the pairs are not identical. Each CounterCount CF image is edited from its factual image and differs only in the counted attribute. This is useful because the two images are otherwise identical, so a change in the model’s answer can be attributed to the counted attribute and the prior, not to differences in view, background, or subject appearance.

To our knowledge, CounterCount is the first richly annotated diagnostic framework for counterfactual counting: it measures the bias, helps diagnose where it comes from, and provides a toolbox to drive research on closing this gap. A comparison using the same VLMs on the existing datasets would only reproduce the accuracy and bias rates already reported there.

3. Generalization to other image generators

We expect our findings to generalize to images from other sources, because the bias depends on the subject being recognized and the edited attribute being clearly visible, not on how the image was produced. We expect this for three reasons:

  • Factual and CF images come from the same pipeline. Yet model accuracy drops from 91.6% to 49.5%. If the generator were the source of failure, factual accuracy would drop as well.
  • Humans count the same images correctly. Every image was manually verified and humans reach 97.5% on the CF images, so the edits are clearly visible. A model that grounds its answer in the image, as humans do, should reach comparable accuracy.
  • Other image sources lead to the same conclusion. VLMBias (Vo et al., ICLR 2026) generated its images with two other generators (Gemini-2.0 Flash and GPT-4o) and concluded that the failure comes from the VLMs themselves, regardless of the generator.

Thanks to the authors for their responses and additional analyses. However, I still have the following concerns:

  1. A key motivation of CounterCount is to reduce the confounding effect of general counting difficulty by focusing on small attribute counts. However, the results still show substantial counting-related effects, and the additional analysis does not fully disentangle these effects from knowledge-prior bias. Therefore, the claimed advantage of CounterCount in isolating prior bias over existing benchmarks remains insufficiently established.
  2. The authors emphasize the broader coverage and richer annotations of CounterCount, but these design advantages do not necessarily translate into improved diagnostic capabilities. The claim that evaluating the same VLMs on existing datasets would “only reproduce the accuracy and bias rates already reported” does not address the central issue. Without a direct empirical comparison or evidence of diagnostic insights unavailable from existing benchmarks, the incremental contribution of CounterCount remains unclear. For these reasons, I do not believe the authors have sufficiently established the claimed contributions, and I will maintain my original rating.

We thank the reviewer for the follow-up. In our previous response, we argued that a comparison on the existing datasets would not change our diagnosis. We decided that this is better shown than argued, so we ran it, together with a new analysis for the first concern.

1. Disentangling counting difficulty from the prior

We agree that counting-related effects exist, mainly at larger counts. However, counting difficulty is not the main issue here; the prior is. Every CF image has a factual image of the same scene and attribute, so we keep only the pairs where the model answers the factual image correctly (all 16 models):

Mean of 16 modelsBias (all pairs)Bias (factual image correct)Miscounts (factual image correct)Pairs kept
MCQ39.1%42.1%9.1%508 / 554
OE39.3%42.1%10.9%504 / 554
  • The bias does not decrease for any of the 16 models, in either MCQ or OE.
  • The errors are not miscounts: 42.1% of answers return exactly the canonical count, against 9–11% for other wrong numbers.
  • Humans count these images correctly: 98.2% on addition and 95.0% on removal, against 55.1% and 29.2% for models.

In each pair, the two images show the same scene and differ only in the count. On the factual image, where the count matches what the model expects, the model answers correctly. On the CF image, where it does not, the model still returns the canonical count. Since the scene and the attribute are the same in both images, the error cannot come from the attribute being hard to count. This analysis needs a factual image for every CF image that differs only in the counted attribute. In VLMBias, only the 91 animal images come with a factual image that differs solely in the edit (one added leg). CDH-Bench pairs are generated separately rather than edited from the same image, so a different answer on a pair can also come from differences in the scene. In CounterCount, each of the 554 CF images is paired with the factual image it was edited from.

2. Direct comparison and insights unavailable from existing datasets

Given the time and compute cost of all 16 models, we evaluated our two strongest open-weight models on the counting images of CDH-Bench (98 pairs, in MCQ format; two pairs that are not counting questions were excluded) and VLMBias (332 images, our OE prompt):

DatasetModelCF accuracyBiasMiscount
CounterCount (MCQ)Qwen3.8-27B62.1%29.1%8.8%
Gemma-4-31B64.6%25.8%9.6%
CDH-Bench (MCQ)Qwen3.8-27B53.1%30.6%16.3%
Gemma-4-31B66.3%21.4%12.2%
CounterCount (OE)Qwen3.8-27B58.5%31.4%10.1%
Gemma-4-31B63.5%24.2%12.3%
VLMBias (OE)Qwen3.8-27B32.8%61.4%5.7%
Gemma-4-31B17.2%74.1%8.7%

All three datasets detect the bias, as prior work already showed. Detecting it is not our goal. We want to be clear about our contribution: this is an analysis paper. CounterCount is the tool we built to carry out this analysis, because the existing datasets were not sufficient for it. It can be used as a benchmark, but our claims are about the diagnosis: when the bias occurs, how strong it is, and whether the model has the visual evidence. These insights can only be obtained from CounterCount:

  • Attention to the counted attribute. Models attend less to the counted region when they fail, and amplifying that attention improves CF accuracy by 3.3 points across 14 models. Neither dataset localizes the counted attribute.
  • The bias follows the strength of the prior. Bugs reach 20% accuracy with 73% bias, while housing reaches 82% with 9% bias (Figure 16). VLMBias only contains subjects with a very strong prior, and CDH-Bench has four counting subcategories of 25 pairs.
  • Removal vs. addition. Removal induces about twice the bias of addition (64.2% vs 32.2%). VLMBias animals and logos only have attributes added, and CDH-Bench has 8 removal pairs.
  • Magnitude of the edit. For additions, bias falls from 36% to 22% as more attributes are added (Figure 6b). In VLMBias, 95.5% of images change by exactly one.
  • Human evaluation on the whole dataset. Humans score 98.9% and 97.5% across all pairs. CDH-Bench reports no human accuracy, and VLMBias only on its animal images.

We hope this addresses the two remaining concerns.

Reviewer_44qXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work presents a new dataset for evaluating whether VLMs are counting based on visual grounding or based on general knowledge. The dataset involves objects with canonical numbers of parts that instead appear with a different number of parts. VLMs are shown to have significantly reduced accuracy when the number of parts differs from the canonical count, whereas human accuracy is relatively unaffected. A number of factors are shown to partially close this gap, including reasoning and techniques for focusing attention on the relevant image region.

Critical strengths
  • The paper is well written and the method is very clearly described. The dataset is well documented.
  • The question that motivates this work, the extent to which VLMs use general knowledge vs. visual grounding, is interesting and understudied.
  • A clear failure mode is identified and several methods are explored for remedying it.
  • A broad range of models are evaluated.
Critical weaknesses
  • This work is motivated by the goal of evaluating the extent to which VLMs rely on visual grounding vs. general knowledge, but the focus on counting, which is notoriously difficult for VLMs without reasoning, presents a confound. It is well known that VLMs struggle with counting and rely on reasoning to perform counting accurately. The paper argues that this shouldn't be an issue for smaller counts, but VLMs often struggle even in this setting. The results also confirm that the gap between factual and counterfactual accuracy grows as a function of the count, indicating that this gap may be driven by underlying difficulty with counting. If the goal is to assess the interplay between visual grounding and general knowledge, it would be better to investigate additional domains where performance is reasonably high in a non-reasoning mode.
  • Although it is good to evaluate a range of open-source models, it would also be informative to know how much frontier models struggle with this task, especially with reasoning enabled.
  • In the experiments with open-source models with reasoning enabled, the paper states that the same prompt was used. Some of the prompts explicitly instruct the model to reply only with a count. In these experiments, did the models actually perform reasoning? How long were the reasoning traces and can some example traces be provided?
Questions for authors
  • How do frontier models with reasoning enabled perform on this task?
  • In the experiments with open-source models with reasoning enabled, was reasoning actually performed? Can example traces and trace length statistics be shared?
  • Can the core question concerning visual grounding vs. general knowledge be explored in other domains?
3 comments

1. Counting difficulty vs. prior bias

We thank the reviewer for the constructive feedback. We address each point below
We agree with the reviewer that VLMs struggle with counting. However, our experimental setup was intentionally designed with factual/counterfactual image pairs with small counts to neutralize the impact of counting difficulty.

Figure 6a in the paper shows that VLMs can count well across all attribute counts. On the other hand, the performance degrades consistently on counterfactual images. This indicates that the high accuracy on factual images is mostly coming from using the knowledge prior to answering questions rather than actually counting attributes. Moreover, if the poor performance is caused by the difficulty of the counting task, the model predictions’ would be spread across random miscounts. Instead, 77% of the predicted miscounts are the canonical counts (Figure 4b) demonstrating that the problem is mostly caused by the model bias, not the model’s inability to count.

2. Counting with and without reasoning

The reviewer suggests studying domains where non-reasoning performance is reasonably high. Our task already meets this condition because, in our experiments, non-reasoning models achieve an average accuracy of 91.6% on factual images (i.e. whenever the image agrees with the prior). This high performance drops significantly to 49.5% on counterfactual images (i.e. when the attribute count conflicts with the language prior). Enabling reasoning does not close this gap. The table below summarizes all comparisons with and without reasoning, from the paper (Section 4.1.4) and from this rebuttal (Appendix C.7.3):

ModelCF acc. without reasoningCF acc. with reasoningΔ Acc.
Qwen3.5-4B44.8%51.1%+6.3 pp
Qwen3.5-27B56.1%61.9%+5.8 pp
Gemma-4-E2B31.0%32.5%+1.5 pp
Gemma-4-31B64.6%66.2%+1.6 pp
Sonnet 547.5%47.5%0.0 pp
GPT-5.6 Luna48.2%51.6%+3.4 pp
Mean (6 models)48.7%51.8%+3.1 pp
Humans97.5%——

From this table, we can conclude that:

  • Reasoning helps only slightly. CF accuracy changes by 0.0 to 6.3 points (3.1 on average).
  • No model comes close to humans. The best model with reasoning reaches 66.2%, against 97.5% for humans.
  • This holds for open-weight and closed-source models alike.

3. Visual grounding vs. general knowledge on other domains

Yes. The conflict between general knowledge and the visual input has already been explored for attributes such as color, background, and context (e.g., WHOOPS! (Bitton-Guetta et al., 2023), VLind-Bench (Lee et al., 2025); see Section 5). We study it for counting because counting remains one of the most challenging tasks for VLMs, even frontier ones Hard Vision, Easy Vision (Rasheed et al., 2026), and because the prior has a single known answer, which makes the bias directly measurable. Existing counterfactual counting datasets are too limited to diagnose it (Appendix A.2), so we built a dataset covering as many categories and attributes as possible to do our analysis.

4. Frontier models with reasoning

We agree with the reviewer, which is why our submission already contained experiments with two frontier models, Sonnet 5 and GPT-5.6 Luna (Figure 4a). Both show a similar prior bias to the open-source models. Following the reviewer’s suggestion, we evaluated these models with reasoning enabled, setting the reasoning effort to the maximum. CF accuracy is unchanged for Sonnet 5 and improves by only 3.4 points for GPT-5.6 Luna (see the table in our response 2), while the bias decreases by at most 1.1 points. Humans remain near ceiling on both factual and CF images (98.9% and 97.5%). In contrast, both models score above 94% on factual images but remain around 50% on CF images, even with reasoning. These results show that a large gap remains even for frontier models and that reasoning does not solve the problem, since the bias persists. We added these results to Appendix C.7.3.

5. Whether reasoning was performed

Yes, we used the same prompt, and the models did perform reasoning before providing their final answer. Reasoning was enabled through each model’s thinking mode, which generates the reasoning trace separately before the final answer. Across the Gemma models, reasoning traces average 208 ± 269 tokens (mean ± std), while the Qwen models reason substantially longer, averaging 1044 ± 1738 tokens. Based on the reviewer’s suggestion, we added these statistics and included a few reasoning trace examples in Appendix C.7.4. Even with reasoning, CF accuracy improves by only 3.8 points on the open-weight models and by at most 3.4 points on the closed-source models. The traces show that the reasoning itself often returns the canonical count.

Thank you to the authors for these responses. I still have the following concerns:

  • I still believe that it is important to investigate other tasks, and that the current focus on counting is too narrow.
  • The authors cite other studies that have investigated similar questions in other task domains. This suggests that the present work is incremental. I also couldn't find some of the referenced studies in the paper.
  • Sonnet 5 and gpt 5.6 luna are not frontier models, they are lightweight propietary models. A test on the strongest available models is important especially given that the dataset is being proposed as a benchmark. For these reasons I will stick with my current rating.

We thank the reviewer for their response.

1. Our focus on counting

Our scope is defined as attribute counting, and we investigate knowledge priors in this realm. Instead of choosing a wide exploration of knowledge biases, we choose to go deep in this domain because counting is a challenging case for the prior. Among the anomaly types in CDH-Bench (Chen et al., 2026), counting is reported as the most persistent, but that work does not investigate why. This is why we built a dataset suitable for diagnosis. In our data, the task is easy for humans (97.5%), yet models return the canonical count in 39.1% of their answers.

2. Relation to studies in other domains

“Cit[ing] other studies that have investigated similar questions in other task domains” does not mean that our contribution is incremental. We cited them only to answer the reviewer’s question on whether the core question can be explored in other domains. Our findings are specific to counting. For example, removing an attribute induces about twice the bias of adding one (64.2% vs 32.2%), and models attend less to the counted attribute when they fail, although the visual evidence is available. If the reviewer has any factual evidence for their claim, we would appreciate their source.

Regarding the references: VLind-Bench (Lee et al., 2025) is cited in Section 5. Our response mentioned WHOOPS! (Bitton-Guetta et al., 2023), while the paper cites WHOOPS-AHA (“When Seeing Overrides Knowing”, Ortu et al., 2025), which builds on it, in Section 5. Rasheed et al. (2026) was released after the submission deadline, so it could not be cited; we will add it to the paper.

3. Frontier models

We also want to be clear about our contribution: this is an analysis paper. CounterCount can be used as a benchmark, but that is not our aim. Our primary aim is to help the open-source community, which is why 14 of our 16 models are open-weight, and our attention analyses need access to model internals.

Following the reviewer’s request, we evaluated Claude Opus 5 and GPT-6 Astra. GPT-6 Astra is a reasoning model, so it is comparable only to the reasoning results:

ModelReasoningFactual accuracyCF accuracyCF bias
Claude Opus 5Off97.5%58.5%33.2%
Claude Opus 5On98.0%63.0%30.1%
GPT-6 AstraOn97.5%89.7%6.9%
Gemma-4-31B (best open-weight, Figure 9)On—65.9%25.5%
Humans—98.9%97.5%—
  • Claude Opus 5 shows the same bias as the open-weight models. It drops from 97.5% to 58.5%, one third of its answers return the canonical count, and reasoning improves it by only 4.5 points.
  • GPT-6 Astra largely overcomes the bias, reaching 89.7% with 6.9% bias, although it remains below humans (97.5%). This agrees with our analysis that reasoning reduces the bias (Section 4.1.4, Figure 9).
  • It takes the largest reasoning model to get there. The best open-weight model with reasoning reaches 65.9%.

Counting at these low counts should not require this much compute. Humans enumerate up to about four to six items without counting, and VLMs are reported to count scenes of one to five objects with very high accuracy when no prior is involved (Campbell et al., 2024, cited in Section 5). In CounterCount, 96.6% of images have a canonical count of four or less.

The models that matter most here are the ones that run where compute is limited. For example, a 2-billion-parameter VLM was recently demonstrated running locally on smart glasses (PrismML and Qualcomm, 2026: https://prismml.com/news/prismml-brings-1-bit-bonsai-models-to-ai-smart-glasses-powered-by-snapdragon). Such a model should count correctly when it sees a person with a missing finger, and removal cases are where we find the strongest bias. We want our dataset and analyses to help the open-source community close this gap.

We hope this addresses the reviewer’s concerns.

Reviewer_2XCSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CounterCount, a diagnostic dataset designed to expose how VLMs rely on knowledge priors rather than genuine visual grounding when counting object attributes. By utilizing paired factual and counterfactual images with small, non-canonical counts, the study demonstrates that while humans maintain near-perfect accuracy, VLMs suffer significant performance drops by defaulting to system-1-like canonical answers.

Critical strengths

The paper is overall easy to follow.

The proposed dataset appears to have high quality.

Critical weaknesses

While the experimental setup is reasonable, the core findings appear largely expected and fail to offer significant novel insights. Furthermore, the paper lacks a clear path toward practical solutions. The proposed modulation methods yield only marginal improvements, leaving the identified issues effectively unresolved.

Questions for authors

N/A

3 comments

Identifying failure modes of vision-language models (VLMs) using counterfactual images has been explored in several previous works, including:

  • Vo, An, et al. "Vision Language Models Are Biased." ICLR, 2026.
  • Golovanevsky, Michal, et al. "Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts." Proceedings of EMNLP, 2025.
  • Lee, Kang-il, et al. "VLind-Bench: Measuring Language Priors in Large Vision-Language Models." Findings of NAACL, 2025.

In particular, Vision Language Models Are Biased (ICLR 2026) presents counterfactual images and evaluates models using counting questions, making its experimental setting particularly similar to the one investigated in this paper. This substantial overlap raises concerns about the novelty of the proposed benchmark.

All three are cited and discussed in our paper (Section 5 and Appendix A.2) and we summarize them again here in case the reviewer missed them. These works show that the bias exists. Our contribution is to diagnose and understand it, and we focus on counting because it is challenging for models and understudied. We also want to highlight that this is an analysis paper. Our dataset can be used as a benchmark, but that is not our aim.

  • VLind-Bench and Pixels Versus Priors do not study counting. The first uses yes/no questions on language priors, and the second studies color and size.
  • VLMBias is the closest work, and our results agree with theirs that the bias exists. As we note in the paper (Section 5), its edits are limited (for animals, one added leg), it includes high-count cases, and several of its tasks rely on simple geometric patterns, which makes it difficult to separate prior-driven failures from perceptual limits. CounterCount provides much more detailed and structured insights on counting specifically.:
    • The bias follows the strength of the prior. Categories with low accuracy have high bias, and categories with high accuracy have low bias (Figure 16): bugs reach 20% accuracy with 73% bias, while housing reaches 82% with 9% bias. VLMBias only contains subjects with a very strong prior, so it cannot show this.
    • Removal induces about twice the bias of addition (64.2% vs 32.2%). In VLMBias, animals and logos only have attributes added.
    • The visual evidence is available but underused. Models attend less to the counted attribute when they fail, and amplifying that attention recovers answers. This needs our region annotations.
    • Humans score 98.9% on factual and 97.5% on counterfactual across all pairs, so the task is not difficult. VLMBias reports a human study on its animal images only.

We also ran a direct comparison with our two strongest open-weight models on the 332 counting images of VLMBias (OE prompt):

DatasetModelCF accuracyBias
CounterCount (OE)Qwen3.8-27B58.5%31.4%
Gemma-4-31B63.5%24.2%
VLMBias (OE)Qwen3.8-27B32.8%61.4%
Gemma-4-31B17.2%74.1%

Both datasets detect the bias, but the numbers are not directly comparable, since VLMBias differs from CounterCount in several ways at once: 95.5% of its images change by exactly one element, and some of its tasks have very high counts (e.g., 32 chess pieces or 50 stars). CounterCount varies the size and the type of the edit (removal/addition) and covers 17 categories, which is what allows the insights above.

I thank the authors for their detailed response and for clarifying the distinctions between their work and existing literature. However, as noted in my initial review, these differences do not appear to offer significant new insights, especially insights on building models with reduced bias. Consequently, I will maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.