- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CounterCount, a diagnostic benchmark designed to investigate whether vision-language models (VLMs) rely on visual evidence or knowledge priors when counting object attributes. To reduce the confounding effects of general counting difficulty and visual ambiguity in existing counterfactual counting benchmarks, the authors construct 554 manually verified factual–counterfactual (CF) image pairs, covering 379 subjects, 228 attribute types, and 17 semantic categories. The dataset also provides region-level annotations, including segmentation masks and bounding boxes. The authors evaluate 16 VLMs alongside human participants. While the models achieve an average accuracy of 91.6% on factual images, their accuracy drops to 49.5% on CF images, with most incorrect responses matching the canonical attribute count. The paper further investigates this behavior through visual context blurring, instructional cueing, thinking mode, attention analysis, and attention modulation. Based on these experiments, the authors suggest that models tend to fall back on knowledge priors after recognizing familiar objects and may underutilize relevant visual evidence.
Critical strengths
- CounterCount provides a well-motivated and valuable diagnostic benchmark for studying the reliance of VLMs on visual evidence versus knowledge priors. When evaluating knowledge-prior bias through attribute counting, model errors may arise from either reliance on prior knowledge or limitations in visual counting ability, making it difficult to distinguish between these factors. CounterCount addresses this challenge by focusing on relatively small attribute counts, constructing controlled factual–CF image pairs, and manually verifying the visibility and correctness of the edited attributes. These design choices help reduce the influence of general counting difficulty and visual ambiguity, enabling a more targeted assessment of how VLMs respond when visual evidence conflicts with canonical knowledge. Furthermore, the dataset covers a diverse range of subjects and attributes and provides localized annotations for fine-grained analysis. These features make CounterCount a useful resource for the community to further investigate visual grounding and knowledge-prior bias in VLMs.
Critical weaknesses
-
The distinction between knowledge-prior bias and general visual counting difficulty could be better substantiated. A central motivation of CounterCount is to reduce the confounding effects of general counting difficulty by focusing on relatively small attribute counts. While this is a reasonable design choice, the paper's results suggest that counting-related factors may still influence the observed performance differences. For example, Figure 6 shows that CF accuracy decreases as the attribute count increases, while attribute addition and removal exhibit substantially different bias rates. The authors already provide useful analyses of these effects, but the results primarily characterize performance differences across conditions rather than disentangle the contributions of counting difficulty and knowledge-prior reliance. A more controlled analysis, for example by comparing bias under matched attribute counts and editing conditions, would help clarify the extent to which the observed failures specifically reflect knowledge-prior bias and further strengthen the benchmark's diagnostic validity.
-
The practical diagnostic advantages over existing benchmarks could be demonstrated more directly.The paper provides a clear comparison with related benchmarks in the Related Work section and Appendix A.2, highlighting CounterCount's broader subject and attribute coverage, controlled image pairs, and localized annotations. These are meaningful design advantages, and the attention-based experiments demonstrate the usefulness of the additional annotations. However, the comparisons primarily focus on dataset characteristics, while the extent to which these advantages translate into improved diagnostic capabilities remains less clear. Related benchmarks, such as Vo et al. (2025b) and CDH-Bench (Chen et al., 2026), also investigate conflicts between visual evidence and prior knowledge, including counterfactual attribute counting. A direct comparison under comparable counting tasks and evaluation settings would help establish whether CounterCount provides additional insights into model behavior or more effectively differentiates models in terms of their reliance on visual evidence and knowledge priors. Such evidence would further strengthen the case for CounterCount as a valuable benchmark for future research.
Questions for authors
-
Distinguishing knowledge-prior bias from counting difficulty. Figure 6 reports the effects of attribute count and editing direction on CF accuracy and canonical-count bias. Since these factors may be correlated, could the authors provide an additional analysis under matched counting and editing conditions? In particular, does the observed knowledge-prior bias remain consistent after controlling for these factors?
-
Incremental diagnostic value over existing benchmarks. The paper clearly demonstrates the design advantages of CounterCount in terms of dataset coverage and localized annotations. Could the authors further clarify its practical diagnostic advantages over existing counterfactual counting benchmarks? For example, would a comparison using the same VLMs and comparable counting tasks reveal additional model behaviors or distinctions that are less apparent in existing benchmarks?
-
Generalizability of the findings. All CounterCount images are generated using Nano-Banana, which facilitates controlled factual–CF construction. However, different image generation or editing methods may produce different visual characteristics. Could the authors discuss how these differences might affect the observed knowledge-prior bias and the extent to which the findings are expected to generalize to counterfactual images from other sources?
4 comments
3. Generalization to other image generators
We expect our findings to generalize to images from other sources, because the bias depends on the subject being recognized and the edited attribute being clearly visible, not on how the image was produced. We expect this for three reasons:
- Factual and CF images come from the same pipeline. Yet model accuracy drops from 91.6% to 49.5%. If the generator were the source of failure, factual accuracy would drop as well.
- Humans count the same images correctly. Every image was manually verified and humans reach 97.5% on the CF images, so the edits are clearly visible. A model that grounds its answer in the image, as humans do, should reach comparable accuracy.
- Other image sources lead to the same conclusion. VLMBias (Vo et al., ICLR 2026) generated its images with two other generators (Gemini-2.0 Flash and GPT-4o) and concluded that the failure comes from the VLMs themselves, regardless of the generator.
Thanks to the authors for their responses and additional analyses. However, I still have the following concerns:
- A key motivation of CounterCount is to reduce the confounding effect of general counting difficulty by focusing on small attribute counts. However, the results still show substantial counting-related effects, and the additional analysis does not fully disentangle these effects from knowledge-prior bias. Therefore, the claimed advantage of CounterCount in isolating prior bias over existing benchmarks remains insufficiently established.
- The authors emphasize the broader coverage and richer annotations of CounterCount, but these design advantages do not necessarily translate into improved diagnostic capabilities. The claim that evaluating the same VLMs on existing datasets would “only reproduce the accuracy and bias rates already reported” does not address the central issue. Without a direct empirical comparison or evidence of diagnostic insights unavailable from existing benchmarks, the incremental contribution of CounterCount remains unclear. For these reasons, I do not believe the authors have sufficiently established the claimed contributions, and I will maintain my original rating.
We thank the reviewer for the follow-up. In our previous response, we argued that a comparison on the existing datasets would not change our diagnosis. We decided that this is better shown than argued, so we ran it, together with a new analysis for the first concern.
1. Disentangling counting difficulty from the prior
We agree that counting-related effects exist, mainly at larger counts. However, counting difficulty is not the main issue here; the prior is. Every CF image has a factual image of the same scene and attribute, so we keep only the pairs where the model answers the factual image correctly (all 16 models):
| Mean of 16 models | Bias (all pairs) | Bias (factual image correct) | Miscounts (factual image correct) | Pairs kept |
|---|---|---|---|---|
| MCQ | 39.1% | 42.1% | 9.1% | 508 / 554 |
| OE | 39.3% | 42.1% | 10.9% | 504 / 554 |
- The bias does not decrease for any of the 16 models, in either MCQ or OE.
- The errors are not miscounts: 42.1% of answers return exactly the canonical count, against 9–11% for other wrong numbers.
- Humans count these images correctly: 98.2% on addition and 95.0% on removal, against 55.1% and 29.2% for models.
In each pair, the two images show the same scene and differ only in the count. On the factual image, where the count matches what the model expects, the model answers correctly. On the CF image, where it does not, the model still returns the canonical count. Since the scene and the attribute are the same in both images, the error cannot come from the attribute being hard to count. This analysis needs a factual image for every CF image that differs only in the counted attribute. In VLMBias, only the 91 animal images come with a factual image that differs solely in the edit (one added leg). CDH-Bench pairs are generated separately rather than edited from the same image, so a different answer on a pair can also come from differences in the scene. In CounterCount, each of the 554 CF images is paired with the factual image it was edited from.
2. Direct comparison and insights unavailable from existing datasets
Given the time and compute cost of all 16 models, we evaluated our two strongest open-weight models on the counting images of CDH-Bench (98 pairs, in MCQ format; two pairs that are not counting questions were excluded) and VLMBias (332 images, our OE prompt):
| Dataset | Model | CF accuracy | Bias | Miscount |
|---|---|---|---|---|
| CounterCount (MCQ) | Qwen3.8-27B | 62.1% | 29.1% | 8.8% |
| Gemma-4-31B | 64.6% | 25.8% | 9.6% | |
| CDH-Bench (MCQ) | Qwen3.8-27B | 53.1% | 30.6% | 16.3% |
| Gemma-4-31B | 66.3% | 21.4% | 12.2% | |
| CounterCount (OE) | Qwen3.8-27B | 58.5% | 31.4% | 10.1% |
| Gemma-4-31B | 63.5% | 24.2% | 12.3% | |
| VLMBias (OE) | Qwen3.8-27B | 32.8% | 61.4% | 5.7% |
| Gemma-4-31B | 17.2% | 74.1% | 8.7% |
All three datasets detect the bias, as prior work already showed. Detecting it is not our goal. We want to be clear about our contribution: this is an analysis paper. CounterCount is the tool we built to carry out this analysis, because the existing datasets were not sufficient for it. It can be used as a benchmark, but our claims are about the diagnosis: when the bias occurs, how strong it is, and whether the model has the visual evidence. These insights can only be obtained from CounterCount:
- Attention to the counted attribute. Models attend less to the counted region when they fail, and amplifying that attention improves CF accuracy by 3.3 points across 14 models. Neither dataset localizes the counted attribute.
- The bias follows the strength of the prior. Bugs reach 20% accuracy with 73% bias, while housing reaches 82% with 9% bias (Figure 16). VLMBias only contains subjects with a very strong prior, and CDH-Bench has four counting subcategories of 25 pairs.
- Removal vs. addition. Removal induces about twice the bias of addition (64.2% vs 32.2%). VLMBias animals and logos only have attributes added, and CDH-Bench has 8 removal pairs.
- Magnitude of the edit. For additions, bias falls from 36% to 22% as more attributes are added (Figure 6b). In VLMBias, 95.5% of images change by exactly one.
- Human evaluation on the whole dataset. Humans score 98.9% and 97.5% across all pairs. CDH-Bench reports no human accuracy, and VLMBias only on its animal images.
We hope this addresses the two remaining concerns.
1. Prior bias under matched counts and edits
We thank the reviewer for their feedback and for recognizing the value of CounterCount. Figure 6a already compares factual and CF accuracy at the same count. The number of attributes to count is identical, yet factual accuracy is higher than CF accuracy at every count, showing that the high factual accuracy reflects prior knowledge rather than actual counting. As for editing conditions, we understand that the reviewer refers to the addition and removal of attributes. Following the reviewer’s suggestion, we further analyzed bias under the same count for the two modifications, as shown in the table below:
The table shows that the bias remains after controlling for both factors. It is present at every count for both edits, and for the same count it is much higher for removal than for addition (e.g., 61.1% vs 23.6% at a count of two). Counting difficulty cannot explain this difference, for three reasons:
2. Diagnostic advantages over existing datasets
We thank the reviewer for recognizing that the region-level annotations and the attention-based experiments they enable are useful. Below, we state concretely what the existing Counterfactual counting datasets , VLMBias (Vo et al., ICLR 2026; cited as Vo et al., 2025b in our submission) and CDH-Bench (Chen et al., 2026), do not allow, and what CounterCount adds:
Limited subjects and attributes. VLMBias has 332 unique counting images covering about 19 attributes in 7 categories. CDH-Bench has 3 categories, of which only one is counting, with 100 counting pairs covering 66 subjects and 60 attributes. CounterCount has 554 pairs covering 379 subjects and 228 attributes in 17 categories.
No fine-grained annotations for diagnosis and mitigation. VLMBias and CDH-Bench do not localize the counted attribute, so they can only evaluate models or test simple mitigations (prompt instructions and background removal in VLMBias). Our verified subject mask and attribute bounding box for every CF image allowed us to measure how much attention models give to the counted region, showing that they under-attend to it when they fail (Section 4.2), and to test two region-targeted mitigation strategies: blurring the visual context while keeping the attribute intact (Section 4.1.2) and modulating attention toward the counted region (Section 4.2). This helps analyze and understand the behavior of VLMs more precisely.
Human Evaluation CDH-Bench does not provide any human evaluation, and VLMBias reports a human study only on animals category. CounterCount provideshuman evaluation to have a solid upper-bound on how far we can push the models.
No controlled factual–CF pairs. VLMBias evaluates unmodified images only as a 66-image sanity check, and CDH-Bench pairs are generated separately so the pairs are not identical. Each CounterCount CF image is edited from its factual image and differs only in the counted attribute. This is useful because the two images are otherwise identical, so a change in the model’s answer can be attributed to the counted attribute and the prior, not to differences in view, background, or subject appearance.
To our knowledge, CounterCount is the first richly annotated diagnostic framework for counterfactual counting: it measures the bias, helps diagnose where it comes from, and provides a toolbox to drive research on closing this gap. A comparison using the same VLMs on the existing datasets would only reproduce the accuracy and bias rates already reported there.