- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PhysCount, a diagnostic benchmark designed to evaluate whether multimodal large language models (MLLMs) can distinguish visual appearances from physical entities. Existing visual counting benchmarks typically treat visible instances as the units to be counted, making it difficult to handle multiple appearances arising from reflections and transparent surfaces. To address this limitation, the authors collect 546 real-world images and construct 651 target-specific annotation groups containing 4,595 human-written questions. By annotating appearance locations, image-formation modes, and physical-identity groups, PhysCount supports four complementary question families: appearance enumeration, entity enumeration, image-formation counting, and correspondence reasoning. The benchmark also introduces evaluation metrics such as complete-group accuracy (Group-Acc). The authors evaluate 40 MLLMs alongside human participants. The results show that models perform substantially better at enumerating visible appearances than at answering questions requiring physical-identity correspondence, with even the strongest model correctly answering every question in only 13.36% of annotation groups. Further experiments investigate the effects of input resolution, appearance multiplicity, cross-mode correspondence, and joint question presentation on model performance.
Critical strengths
-
The paper identifies a meaningful distinction in visual counting and introduces a benchmark with valuable diagnostic capabilities for the research community. The paper highlights that visual appearances and physical entities are not always in one-to-one correspondence in scenes involving reflections and transparent surfaces, whereas existing visual counting benchmarks generally do not explicitly distinguish between these counting units. To address this issue, PhysCount organizes appearance enumeration, entity enumeration, image-formation counting, and correspondence reasoning into four complementary question families supported by shared structured annotations. This design allows the benchmark to evaluate not only whether models can correctly enumerate visible appearances, but also whether they can understand the relationships between different appearances and the physical entities they depict. As a result, PhysCount provides a valuable evaluation resource for future research on visual counting and physical scene understanding.
-
The dataset construction is systematic, and the paper provides comprehensive evaluations and informative diagnostic analyses. PhysCount is constructed from real-world images through a multi-stage process involving human annotation, verification, and question–answer consistency checks. The authors evaluate 40 MLLMs and establish a human baseline using the same independent-questioning protocol, clearly demonstrating the performance gap between appearance enumeration and counting tasks that depend on physical-identity correspondence. In addition, the paper systematically examines the effects of input resolution, appearance multiplicity, cross-mode correspondence, and joint question presentation. These experiments provide useful insights into how scene structure and evaluation settings affect model performance. The results demonstrate the diagnostic value of PhysCount and offer useful references for future research on physical-entity perception and identity correspondence in MLLMs.
Critical weaknesses
-
The current evaluation does not sufficiently disentangle errors in physical-identity correspondence from errors in subsequent semantic understanding and logical operations. PhysCount demonstrates a clear performance gap between appearance enumeration and counting tasks that depend on physical-identity correspondence through its four question families. However, these question families differ not only in their visual requirements but also in their demands on language understanding and logical operations. For example, correspondence reasoning requires models to establish identity relationships across appearances and perform operations involving set intersections, exclusions, and multiplicities. The examples in Appendix D further show that models may correctly describe the scene and identify the relevant entities but still produce incorrect answers because they fail to perform the required set operations. Although the authors acknowledge these different error sources and provide qualitative analyses, the current evaluation primarily relies on final count predictions and does not systematically disentangle the contributions of appearance detection, physical-identity matching, and subsequent logical operations to the observed errors. Consequently, while PhysCount effectively identifies which types of counting questions models struggle with, its ability to diagnose the specific stages at which failures occur remains limited. Given the rich structured annotations already available in the dataset, incorporating more fine-grained or stage-wise evaluations could further strengthen its diagnostic capabilities.
-
Some supplementary experiments are conducted on a limited set of models, leaving the generality of their findings insufficiently established. Although the main evaluation covers 40 MLLMs, the input-resolution and joint question presentation experiments are primarily conducted using Qwen3.8-27B. For example, Table 3 shows that presenting related questions from the same annotation group jointly substantially improves both Score and Group-Acc, suggesting that question presentation can have an important effect on counting tasks involving physical entities. However, it remains unclear whether this behavior generalizes to other models and whether similar improvements can be consistently observed across different model families. Moreover, joint question presentation also changes the task context available to the model. The observed improvements may result from complementary information provided by related questions, but they may also reflect changes in prompting or multi-question reasoning behavior. Extending these experiments to more representative models and further examining the conditions under which joint question presentation improves performance would strengthen the generality and interpretation of these findings.
Questions for authors
-
Distinguishing different sources of counting errors. The correspondence reasoning tasks in PhysCount involve both physical-identity matching and set operations. Given that Appendix D provides examples where models correctly interpret the scene but fail in subsequent logical operations, could the authors further analyze the relative contributions of these error sources? For example, could the existing structured annotations be used to evaluate models' ability to perform the required counting operations when ground-truth appearance information or physical-identity correspondences are explicitly provided?
-
Generality of the joint question presentation results. Table 3 shows that joint question presentation substantially improves the overall performance of Qwen3.8-27B. Could the authors extend this experiment to other representative models to examine whether the improvement generalizes across model families? In addition, could the authors further discuss whether the gains primarily arise from complementary information shared across related questions or from changes in the models' reasoning behavior under different question presentation settings?