acceptodds
Under review as a conference paper at ICLR 2027

PhysCount: Do MLLMs Count Visual Appearances or Physical Entities?

Abstract

Images reveal appearances, but counting often asks about physical objects. A single entity may be visible directly, in several reflections, or through glass, so detecting every visible occurrence does not determine how many entities are present. Existing visual counting benchmarks rarely separate these two operations. We introduce PhysCount, a benchmark for evaluating whether multimodal large language models (MLLMs) can distinguish what appears in an image from what physically exists in the scene. PhysCount contains 546 real-world images and 4,595 human-written questions organized into 651 target-specific groups. For each group, annotators locate the relevant appearances, label how each is observed, and identify appearances of the same entity. Four complementary question families test the enumeration of appearances, the enumeration of physical entities, image-formation-specific counting, and appearance–entity correspondence. We measure both accuracy on each family and the proportion of groups answered entirely correctly. Across 40 MLLMs, counting visible appearances is substantially easier than recovering the identities that connect them. Even the strongest model answers every question correctly in only 13.36% of groups. Performance declines when one entity produces many appearances or when its appearances cross observation modes. Higher image resolution improves access to visual evidence but brings limited gains in complete-group accuracy, whereas answering related questions jointly produces a marked improvement. These results expose a gap between recognizing what is visible and correctly recovering the physical entities behind those observations.

Primary area:
datasets and benchmarks
Keywords:
Visual Counting, Multimodal Large Language Models, Visual Reasoning, Mirror Reflection, Benchmark Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4rEfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PhysCount, a diagnostic benchmark designed to evaluate whether multimodal large language models (MLLMs) can distinguish visual appearances from physical entities. Existing visual counting benchmarks typically treat visible instances as the units to be counted, making it difficult to handle multiple appearances arising from reflections and transparent surfaces. To address this limitation, the authors collect 546 real-world images and construct 651 target-specific annotation groups containing 4,595 human-written questions. By annotating appearance locations, image-formation modes, and physical-identity groups, PhysCount supports four complementary question families: appearance enumeration, entity enumeration, image-formation counting, and correspondence reasoning. The benchmark also introduces evaluation metrics such as complete-group accuracy (Group-Acc). The authors evaluate 40 MLLMs alongside human participants. The results show that models perform substantially better at enumerating visible appearances than at answering questions requiring physical-identity correspondence, with even the strongest model correctly answering every question in only 13.36% of annotation groups. Further experiments investigate the effects of input resolution, appearance multiplicity, cross-mode correspondence, and joint question presentation on model performance.

Critical strengths
  1. The paper identifies a meaningful distinction in visual counting and introduces a benchmark with valuable diagnostic capabilities for the research community. The paper highlights that visual appearances and physical entities are not always in one-to-one correspondence in scenes involving reflections and transparent surfaces, whereas existing visual counting benchmarks generally do not explicitly distinguish between these counting units. To address this issue, PhysCount organizes appearance enumeration, entity enumeration, image-formation counting, and correspondence reasoning into four complementary question families supported by shared structured annotations. This design allows the benchmark to evaluate not only whether models can correctly enumerate visible appearances, but also whether they can understand the relationships between different appearances and the physical entities they depict. As a result, PhysCount provides a valuable evaluation resource for future research on visual counting and physical scene understanding.

  2. The dataset construction is systematic, and the paper provides comprehensive evaluations and informative diagnostic analyses. PhysCount is constructed from real-world images through a multi-stage process involving human annotation, verification, and question–answer consistency checks. The authors evaluate 40 MLLMs and establish a human baseline using the same independent-questioning protocol, clearly demonstrating the performance gap between appearance enumeration and counting tasks that depend on physical-identity correspondence. In addition, the paper systematically examines the effects of input resolution, appearance multiplicity, cross-mode correspondence, and joint question presentation. These experiments provide useful insights into how scene structure and evaluation settings affect model performance. The results demonstrate the diagnostic value of PhysCount and offer useful references for future research on physical-entity perception and identity correspondence in MLLMs.

Critical weaknesses
  1. The current evaluation does not sufficiently disentangle errors in physical-identity correspondence from errors in subsequent semantic understanding and logical operations. PhysCount demonstrates a clear performance gap between appearance enumeration and counting tasks that depend on physical-identity correspondence through its four question families. However, these question families differ not only in their visual requirements but also in their demands on language understanding and logical operations. For example, correspondence reasoning requires models to establish identity relationships across appearances and perform operations involving set intersections, exclusions, and multiplicities. The examples in Appendix D further show that models may correctly describe the scene and identify the relevant entities but still produce incorrect answers because they fail to perform the required set operations. Although the authors acknowledge these different error sources and provide qualitative analyses, the current evaluation primarily relies on final count predictions and does not systematically disentangle the contributions of appearance detection, physical-identity matching, and subsequent logical operations to the observed errors. Consequently, while PhysCount effectively identifies which types of counting questions models struggle with, its ability to diagnose the specific stages at which failures occur remains limited. Given the rich structured annotations already available in the dataset, incorporating more fine-grained or stage-wise evaluations could further strengthen its diagnostic capabilities.

  2. Some supplementary experiments are conducted on a limited set of models, leaving the generality of their findings insufficiently established. Although the main evaluation covers 40 MLLMs, the input-resolution and joint question presentation experiments are primarily conducted using Qwen3.8-27B. For example, Table 3 shows that presenting related questions from the same annotation group jointly substantially improves both Score and Group-Acc, suggesting that question presentation can have an important effect on counting tasks involving physical entities. However, it remains unclear whether this behavior generalizes to other models and whether similar improvements can be consistently observed across different model families. Moreover, joint question presentation also changes the task context available to the model. The observed improvements may result from complementary information provided by related questions, but they may also reflect changes in prompting or multi-question reasoning behavior. Extending these experiments to more representative models and further examining the conditions under which joint question presentation improves performance would strengthen the generality and interpretation of these findings.

Questions for authors
  1. Distinguishing different sources of counting errors. The correspondence reasoning tasks in PhysCount involve both physical-identity matching and set operations. Given that Appendix D provides examples where models correctly interpret the scene but fail in subsequent logical operations, could the authors further analyze the relative contributions of these error sources? For example, could the existing structured annotations be used to evaluate models' ability to perform the required counting operations when ground-truth appearance information or physical-identity correspondences are explicitly provided?

  2. Generality of the joint question presentation results. Table 3 shows that joint question presentation substantially improves the overall performance of Qwen3.8-27B. Could the authors extend this experiment to other representative models to examine whether the improvement generalizes across model families? In addition, could the authors further discuss whether the gains primarily arise from complementary information shared across related questions or from changes in the models' reasoning behavior under different question presentation settings?

Reviewer_U4LsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PhysCount, a benchmark designed to evaluate whether MLLMs can distinguish visual appearances from physical entities in scenes involving reflections and transparent surfaces, where an object may appear multiple times or be visible only through its reflection. The benchmark contains 546 real-world images and 4,595 questions covering appearance counting, physical entity counting, image formation, and correspondence reasoning. Experiments across 40 MLLMs show that models struggle to consistently identify physical entities across different visual appearances, with the best-performing model achieving only 13.36% complete-group accuracy. The authors further analyze the effects of image resolution, appearance multiplicity, observation modes, and joint question presentation on model performance.

Critical strengths

The authors evaluate 40 MLLMs, including proprietary and open-weight models, along with a human baseline. This provides a broad assessment of current model performance and highlights a substantial gap between models and humans.

The benchmark includes detailed annotations of visual appearances, observation modes, and physical-entity correspondences, enabling the construction of multiple related counting questions from the same scene.

Critical weaknesses

While the paper introduces a new benchmark for distinguishing visual appearances from physical entities, it is unclear what new insights this benchmark provides about MLLMs beyond their already known limitations in visual counting and reasoning. The main findings, including difficulties on questions requiring appearance–entity correspondence, performance degradation with increasing appearance multiplicity, and limited improvements in complete-group accuracy from higher image resolution, are largely expected. Also, the paper does not sufficiently establish how these findings extend our understanding of MLLM capabilities beyond existing benchmarks. So the added scientific value of the proposed benchmark remains unclear.

Although the benchmark evaluates different counting tasks, the evaluation relies on final numerical answers, making it difficult to distinguish errors in object detection, reflection recognition, identity matching, and numerical reasoning. Without more controlled experiments isolating these factors, it is unclear whether the observed failures reflect a distinct limitation in physical-entity understanding or more general visual perception and reasoning difficulties.

Questions for authors

Does this benchmark expose failures in models that otherwise perform well on standard visual counting tasks? Does PhysCount reveal substantially different model rankings or failure patterns that are not captured by existing benchmarks?

Reviewer_qEmbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PhysCount, a benchmark that evaluates whether MLLMs can distinguish visual appearances from physical entities in images involving reflections and transparent surfaces. It contains 546 images and 4,595 questions covering appearance counting, entity counting, image formation, and correspondence reasoning. Experiments on 40 MLLMs show that models struggle with counting tasks involving physical identity, especially when objects appear multiple times or through different observation modes.

Critical strengths

Clear motivation and meaningful task design. The distinction between visual appearances and physical entities is important but has received limited attention in existing counting benchmarks. The structured annotations and four question types provide a useful way to study this problem.

Extensive experiments and useful findings. The evaluation covers 40 MLLMs and includes a human baseline. The analyses of multiple reflections, observation modes, and question presentation provide useful insights into current model limitations.

Critical weaknesses
  1. Serious concerns about image provenance and dataset validity. The image in Figure 1 appears to be AI-generated or heavily synthesized, but it is presented without identifying it as synthetic or merely illustrative. This is concerning because the paper explicitly describes PhysCount as a benchmark of real-world or natural images. In this task, physical consistency is essential: AI-generated scenes may contain implausible reflections, inconsistent viewpoints, or objects whose correspondence to a real physical entity is not well defined. If the image is part of the benchmark, its annotations and ground-truth physical counts may not be trustworthy. Even if it is only an illustration, the authors should clearly disclose this and demonstrate that the actual evaluation images are authentic and physically consistent. Without image provenance and a dataset-level audit, I am not confident in the quality and reliability of the benchmark.
  2. Limited analysis of the actual error sources. The evaluation mainly relies on final numerical answers. Incorrect answers may result from missed appearances, incorrect identity matching, or mistakes in understanding the question. It is therefore difficult to determine how much of the performance gap is specifically caused by physical identity reasoning.
  3. Some aspects of data reliability and experimental analysis need further support. Although the paper describes a careful annotation process, it does not report agreement between annotators on physical entity identities. In addition, the comparison between observation modes involves substantially different group sizes, which may affect the reliability of the findings.
Questions for authors
  1. Could the authors provide more details on the quality control process? In particular, were QA pairs accepted by GPT-5.6 Sol randomly checked by human annotators, and what proportion of flagged cases required correction?

  2. Could the authors provide a small analysis to distinguish errors in appearance detection, identity matching, and set operations? This would help clarify which factors contribute most to the observed performance gap.

  3. In Figure 6, the numbers of within mode and cross mode groups appear to be quite different. Could the authors clarify the sample sizes used and provide uncertainty estimates for this comparison?

  4. Gemini 3.7 Flash achieves the best overall performance among the evaluated models. Why was Gemini 3.8 Flash not included, especially since GPT-6 Astra was evaluated? Given the strong results of Gemini 3.7 Flash, could Gemini 3.8 Flash perform even better? Could the authors clarify the model selection criteria and evaluation dates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.