acceptodds
Under review as a conference paper at ICLR 2027

KidVis: Benchmarking Fundamental Visual Primitives in Multimodal Large Language Models

Abstract

Despite the remarkable proficiency of Multimodal Large Language Models (MLLMs) in high-level reasoning, it remains questionable whether they possess human-like foundational visual abilities. To investigate this question, we present KidVis, a diagnostic benchmark grounded in cognitive developmental psychology rather than standard semantic recognition. KidVis deconstructs visual perception into six atomic primitives, including Visual Closure, Spatial Orientation, and Visual Tracking, and instantiates them through controlled, low-semantic, motor-free tasks. We evaluate 22 leading MLLMs against an age-specific human reference baseline. Our study uncovers a fundamental disconnect: while children achieve near-perfect accuracy of 95.31%, the strongest evaluated model, GPT-5, reaches 67.33%. We further observe a visual scaling paradox: within the evaluated model families, larger parameter counts do not consistently translate into stronger foundational visual perception, and may even coincide with semantic interference on low-level tasks. These findings suggest that current MLLMs still face systematic limitations in basic visual grounding, highlighting the need for evaluation and model design beyond semantic-heavy benchmarks and simple parameter scaling.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Model Evaluaiton, Fundamental visual perception

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WVL3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper utilizes the assessment systems from developmental psychology, including Carroll's Three-Stratum Theory of Cognitive Abilities and the Motor-Free Visual Perception Test (MVPT-3), to construct a benchmark KidVis for evaluating the perceptual capabilities of MLLMs on fundamental visual primitives. It defines six atomic visual primitives and maps them in pairs to 10 tasks, with each task consisting of 50 questions, totaling 500 questions. A zero-shot evaluation is conducted on 22 MLLMs, and the experimental results show that the composite score of ten 6–7-year-old children reached 95.31, while the average score of MLLMs is only 38.67. Even the strongest evaluated model, GPT-5, scores only 67.33, far below human children. This benchmark reveals that although MLLMs have demonstrated strong abilities in high-level semantic reasoning, they remain fragile on low-level visual tasks, and this weakness is not consistently alleviated by increasing model parameters.

Critical strengths
  1. The results of the 10 human children are highly valuable. Compared with this baseline, the weakness of the models is shown more clearly, and such a comparison is relatively rare in similar work.
  2. The experimental design is reasonable, and the result analysis is solid, including sensitivity analysis and confidence intervals.
  3. The writing is clear, and the task examples and failure cases are also comprehensive and easy to understand.
Critical weaknesses
  1. The related work is severely lacking. Table 1 only compares five general benchmarks, including MME, MMBench, MMMU, MathVista, and HallusionBench. It highlights the novelty of KidVis in terms of isolated perception, primitive decomposition, child-level, and low semantic. However, these four aspects already have multiple existing works. The concurrent work BabyVision (ICML 2026) [1] also takes the development of children's visual perception as its framework, and tasks such as maze tracing, block counting, and pattern completion heavily overlap with those in this paper. In 2025, VisFactor [2], Do You See Me [3], and Percept-V [4] directly build low-semantic perception benchmarks based on standardized visual perception tests such as FRCT, TVPS, and MVPT, the same tests this paper builds on. In 2024, CoreCognition [5] evaluates 230 models based on developmental cognitive science and reports that low-level abilities do not scale with parameter size. Among these, this paper only cites CoreCognition and merely mentions it in passing as a study of hallucination.
  2. The evidence for the scaling paradox is insufficient. The differences among the three sizes of Qwen3-VL (4B/8B/30B) are less than 5 points, making it difficult to say that there is a real difference. Moreover, the 30B version is MoE with approximately 3B active parameters, and it should not be compared based on the total parameter count. Also, the range of sizes is not large enough and does not cover the dense 32B and MoE 235B versions.
  3. Section 5.5 contains many mechanistic explanations, including that MLLMs lack an inherent inhibition mechanism to suppress background distractors, that ViT patch discretization fragments continuous paths into discrete visual tokens, and that MLLMs treat visual history as a static read-only buffer. None of these is supported by any experiment in the paper.

[1] Liang Chen et al. BabyVision: Visual Reasoning Beyond Language. ICML 2026. https://openreview.net/forum?id=ZD3BniFPto

[2] Jen-Tse Huang et al. Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs. arXiv:2502.16435, 2025. https://arxiv.org/abs/2502.16435

[3] Aditya Sanjiv Kanade and Tanuja Ganu. Do You See Me: A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs. EACL 2026. https://aclanthology.org/2026.eacl-long.343/

[4] Samrajnee Ghosh et al. The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems? COLM 2026. https://arxiv.org/abs/2508.21143

[5] Yijiang Li et al. Core Knowledge Deficits in Multi-Modal Language Models. ICML 2025. https://arxiv.org/abs/2410.10855

Questions for authors
  1. Could you expand the related work and discuss the differences between this paper and those works, in particular the works mentioned in weakness 1? This is important for judging the novelty.
  2. For weakness 2, could you provide the results of Qwen3-VL 32B and 235B? Does the non-monotonic trend still hold?
  3. For weakness 3, could you provide at least one piece of evidence to support these mechanistic explanations?
Reviewer_h8YbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

KidVis evaluates foundational visual perception through 500 questions spanning ten task types and six capability categories. Across 22 MLLMs, it finds substantial gaps relative to a reference group of children aged 6–7, particularly on visual tracking. The paper supplements the benchmark with reasoning-trace analysis and controls examining the effects of prompts, resolution, and path-color cues.

Critical strengths
  1. The benchmark targets fine-grained visual observation. The task construction in Section 3.2 includes unusual anatomy, intersecting paths, and precise spatial relations. Section 5.5 documents cases where plausible semantic expectations conflict with the visible image content.
  2. The analysis extends beyond aggregate model rankings. The child reference group and prompt, resolution, and color-cue controls provide useful context for the results. The 300 annotated traces in Appendix C further characterize observed failures, with reported inter-annotator agreement of κ = 0.809.
Critical weaknesses
  1. Shared tasks complicate the capability-correlation interpretation. Table 2 assigns Schulte Grid and Path Tracing to both Concentration and Tracking, accounting for two of each capability’s three tasks. Their correlation therefore partly reflects overlapping score construction rather than independent evidence of a common perceptual bottleneck. Question: Please report correlations after excluding shared tasks and distinguish them from correlations between the original aggregate scores.
  2. The color intervention does not rule out endpoint matching. Section 5.4 establishes that coloring paths improves accuracy, but does not show whether endpoint colors alone suffice for the improved items. This supports the usefulness of identity cues more directly than continuous tracking through intersections. Question: Please show representative colored items and evaluate a control that neutralizes endpoint-color matching while retaining color cues along the paths.
  3. The architectural explanations exceed the diagnostic evidence. Section 5.5 attributes errors to a lack of attentional inhibition and an internal reference frame, but task accuracy and the trace annotations do not directly measure those internal mechanisms. The observations identify failures, yet do not distinguish these explanations from alternative encoding or reasoning errors. Question: Please qualify untested architectural explanations as hypotheses and identify the evidence needed to distinguish them.
Questions for authors

see weakness

Reviewer_1c9BOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

KidVis is an evaluation benchmark designed to test whether Multimodal Large Language Models (MLLMs) possess foundational, human-like visual perceptual abilities. Grounded in cognitive science (like Carroll’s Three-Stratum Theory of Cognitive Capabilities and MVPT-3 frameworks), the authors decompose visual perception into six atomic primitives: Visual Concentration (VC), Visual Tracking (VT), Visual Discrimination (VD), Visual Memory (VM), Visual Spatial (VS), and Visual Closure (VCl). These six primitives are instantiated across 10 controlled tasks (50 items each, totaling 500 multiple-choice items) used to evaluate both proprietary and open-source MLLMs, compared against age-specific human baseline.

Critical strengths
  • The paper is thorough in experimental controls (95% bootstrap CI, balanced answer option distribution, resolution/prompt sensitivity etc.)
  • The paper provides some insightful ablations.
Critical weaknesses
  • Overstated Novelty and Missing prior work:

    • My primary concern is that the paper misses some highly relevant works completely, that tackle the same research question, with similar cognitive frameworks and task paradigms, like:

      • Percept-V (Ghosh et. el: https://arxiv.org/pdf/2508.21143 ) which follows the TVPS-4 framework to generate 6000 samples from 30 domains (along 7 skills, several of which overlap with KidVis) with controllable complexities with no manual effort for answer validation and fine tuning analysis, including similar resolution/prompt/answer-type sensitivity ablations.
      • BabyVision (Chen et. al https://arxiv.org/pdf/2601.06521 ) which additionally tests children aged 3,6, 10 and 12 and has substantial task overlap with KidVis and anticipates several of the main findings.
    • In Section 1 and Table 1, the authors claim that "existing works often fail to systematically decompose vision into constitutive cognitive primitives or benchmark these deficits against the foundational stages of human visual development." This doesn't hold true and the authors should explicitly compare and position their work against the existing works as cited above on perception-focused benchmarks and not with general purpose benchmarks like MMMU.

    • Lack of results on fine-tuning MLLMs as done in similar works cited above.

  • Many tasks are not “low-semantic” by design:

    • Body Part Counting, Visual Completion, Complex Scene Counting use naturalistic and AI-generated, scenes of animals, hands, people and household objects. Section 5.5 then points the errors on "semantic prior bias", which is only possible because the items carry semantics.
Questions for authors

Please see Weakness Section

  • In Section 5.3 (Figure 6), the authors report high Pearson correlations across the six capability scores and conclude that "MLLMs tend to improve or fail jointly across perceptual dimensions." However, each Capability Score is a weighted average of 3 to 5 tasks (Table 2). Does this inflate the correlation scores, and is this considered within the experiment?

  • The open source models presented are primarily instruct variants and thinking models should be included to compare against proprietary counterparts

  • Analysis or Discussion on what may be possible approaches to unlock better performance on these tasks would be an interesting addition to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.