acceptodds
Withdrawn from ICLR 2027

HumanPhys: A Human-Centric Physical-World Perception Benchmark for Real-World Visual Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

We introduce HumanPhys, a large-scale benchmark and training corpus for evaluating and improving human-centric physical-world perception in multimodal large language models (MLLMs). While recent multimodal systems have made rapid progress on visual understanding, they still struggle with basic physical reasoning about humans in 3D scenes, such as spatial relations, motion, and deformation. Existing benchmarks capture only a limited subset of these abilities. HumanPhys is designed to systematically probe six dimensions of human-centric physical perception: spatial state, kinematics, behavior, deformation, light-shadow reasoning, and camera-pose trajectory. These dimensions are instantiated as thirteen subtasks: depth estimation, distance estimation, displacement estimation, speed estimation, trajectory understanding, gaze estimation, person counting, blink counting, facial deformation recognition, body-shadow judgment, facial-shadow judgment, environment-centric camera-pose estimation, and human-centric camera-pose estimation. To support both evaluation and model post-training, we further construct an accompanying training corpus of approximately 760K samples. A key challenge in building such a benchmark is obtaining supervision that is both scalable and physically grounded. To this end, we combine curated real-world datasets—JRDB, UCSD, DanceTrack, Eyeblink8, VideoAttentionTarget, and Decaf—with large-scale simulation in Blender with ReplicaCAD. This hybrid pipeline enables clean supervision over diverse perceptual phenomena that are difficult to annotate consistently in the wild. We benchmark a range of open- and closed-source MLLMs on HumanPhys and find that the benchmark remains challenging across all six capability dimensions. We hope HumanPhys will support future research on multimodal systems for human-centered real-world perception and interaction.

Primary area:
datasets and benchmarks
Keywords:
Physical-world perception, Human-centric perception, Spatial intelligence, Multimodal large language models, Benchmark, Simulation-based data synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3qggOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a benchmark (HumanPhys) and a corresponding corpus for evaluating human-centered physical perception in MLLMs. It covers 13 subtasks across six dimensions: spatial state, kinematics, blink events, deformation, illumination, and camera pose. The authors also evaluate multiple MLLMs and fine-tune Qwen3-VL-4B-Instruct on the proposed corpus.

Critical strengths

None

Critical weaknesses
  1. The abstract on the ICLR page is inconsistent with the abstract in the official submission, and the two claims conflict. The page states that "we further construct an accompanying training corpus of approximately 760K samples", whereas the main text states that "together with a 386K-sample training corpus generated by the same pipeline". The huge discrepancy of this magnitude raises doubts about the paper's credibility.
  2. The motivation is questionable. The authors build the benchmark because existing work lacks coverage of human-centered physical quantities in scenes, but the Introduction does not make it clear why evaluating these events is valuable. The authors should justify why MLLMs need to understand such events (e.g., blink counts and illumination on the face).
  3. The strength of MLLMs lies in complex linguistic reasoning or multi-step abstract reasoning. However, the proposed benchmark mainly targets the ability to recover physical quantities from vision (e.g., distance or trajectory), which makes it unlikely to be adopted as a benchmark for mainstream MLLMs. In other words, one would rather evaluate MLLMs on spatial reasoning than on counting blinks.
  4. As a dataset paper, it should at least present a case study for each sub-task to help readers build intuition. However, only Fig. 5 shows one of the 13 sub-tasks. It is also unclear why a small sample of the benchmark data was not included in the supplementary material.
Questions for authors

Response to the questions in the weaknesses.

Reviewer_SnuvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HumanPhys, a benchmark designed to evaluate the human-centric physical perception capabilities of MLLMs. Unlike existing spatial reasoning benchmarks that mainly focus on the geometric relationships between objects and scenes, this work focuses on the ability to comprehend quantitative physical information including human distance, gaze, movement speed, trajectories, blinking, body deformation, lighting, and camera motion. The proposed benchmark is composed of 6 evaluation domains and 13 specific tasks, generating 2,500 evaluation questions and approximately 386,000 training samples by integrating real-world video data with simulations. Experiments conducted on 8 different MLLMs (2 for commercial models and 6 for open-weight models) revealed that even the best-performing model achieved a score of only 46.94 (GPT-5.6-Sol), indicating that current models possess limited human-centric physical perception. Furthermore, by fine-tuning the Qwen3-VL-4B model on our proposed dataset, we demonstrated a significant performance improvement.

Critical strengths
  • Based on the previous benchmarks dealing with the spatial reasoning focused on the object and scene understanding with geometric perspective, the manuscript proposes a novel benchmarks, HumanPhys that extended the quantitative physical information such as spatial states, motion, physiological events and so on. It provides the systematic evaluation framework and helps to analyze the existing MLLM's performance on physical and embodied intelligence.
  • This manuscript introduces a organized data generation pipeline that derives ground-truth answers from sensor metadata. The resulting benchmark and approximately 386K training samples constitute a potentially valuable research resource. In addition, the fine-tuning experiments demonstrate substantial improvements on HumanPhys and selected external spatial reasoning tasks.
Critical weaknesses
  • Section 3 on page 4 of the paper states that while the training and evaluation datasets are partitioned at the question level, they are not separated by scene. Consequently, the same scene may appear in both sets. This data-splitting approach complicates the interpretation of the significant performance improvement observed in models fine-tuned on the HumanPhys dataset. In particular, since the authors did not report the extent of scene overlap, it remains unclear whether these performance gains would persist in novel visual environments. This scenario makes it is difficult to conclude that the model’s generalization capability to unseen or data sources has been sufficiently validated.
  • There is a lack of validation regarding how accurately HumanPhys assesses actual visual and physical reasoning capabilities. As shown in Table 3, text-only models achieved scores of 54.03 on "Trajectory" and 54.19 on "LS-Body." This suggests that, for certain tasks, high scores could potentially be attained based solely on the format of the questions or biases in the distribution of correct answers. Furthermore, human evaluation was conducted on a non-uniform subset of the total questions, making direct comparisons with the models difficult. Consequently, further analysis is required to examine potential biases in the distribution of correct answers and the actual contribution of visual inputs. These limitations raise questions about the evaluation validity of the proposed benchmark.
Questions for authors
  • Although the manuscript states that question-level overlaps between the training and evaluation datasets were removed, it also notes that the same scenes may be present in both sets. Given this, it is difficult to determine whether the significant performance improvement observed after fine-tuning with HumanPhys (from 27.27 to 84.92 points) reflects a genuine enhancement in general physical perception capabilities, or merely an adaptation to specific scenes or data generation patterns.
  • Provide specific details on the extent of overlap, at the level of scenes, individuals, or source videos, between the training and evaluation datasets? Additionally, are there any evaluation results obtained under conditions where scenes or data sources are kept entirely separate?
Reviewer_eeHoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HumanPhys, a benchmark and training dataset for quantitative human-centric physical perception in multimodal large language models. It includes 13 subtasks covering spatial state, kinematics, blink events, deformation, illumination, and camera motion. The authors construct 2,500 evaluation questions and about 386K training examples from existing datasets and rendered scenes. They evaluate eight models and fine-tune Qwen3-VL-4B using HumanPhys alone or together with other spatial datasets. The results show improvements on HumanPhys and external depth and distance tasks. The paper also examines the loss of general spatial abilities after HumanPhys training and how mixed training can reduce this loss.

Critical strengths
  1. The training data shows practical value on external depth and distance tasks. HumanPhys-only training improves four SPAR-Bench subtasks by about 21–45 percentage points. Because these experiments do not use the additional spatial training datasets, they provide direct evidence that HumanPhys data is useful beyond its own benchmark. This is the most convincing result to me, although the evidence mainly supports the depth and distance tasks.
  2. The overall direction is reasonable, and the data construction has practical value. Understanding people through quantitative properties such as distance, motion, and contact response is a useful direction beyond action and behavior recognition. The paper provides an organizing framework and a programmatic pipeline for constructing evaluation and training data. Using source metadata and renderer states reduces the need to write new answers manually and supports larger-scale construction.
Critical weaknesses
  1. The distinctive tasks need stronger validation. Gaze converts existing fixation annotations into an image-plane angle, but its added value over gaze-target prediction is unclear. Deform selects the DECAF reference among rescaled alternatives, without establishing why it is uniquely physically valid or whether visual naturalness and generation artifacts provide shortcuts. The human study, conducted on a non-uniform subset, does not resolve these concerns. Since the external gains mainly support depth and distance, the tasks contributing much of the paper’s novelty have less direct validation.
  2. The evaluation leaves important questions about generalization and attribution. The question-level split allows shared scenes, limiting what the internal training gains show about unseen-scene generalization, although the external depth and distance gains remain valuable. Mixed training lacks a matched baseline using only SpaceR/SpatialCorpus, so HumanPhys’s additional contribution is unclear. Subtask scores exclude unparseable answers while the overall score counts them as zero, making capability comparisons depend on different sample subsets. The claimed human-presence penalty also needs camera paths and scoring rules to be controlled. These issues limit the interpretation of the reported gains and diagnostic findings.
Questions for authors
  1. Clarification and additional evidence that could change my assessment: What overlap exists between training and evaluation at the scene, source-video, temporal-window, and frame levels? Are results available under a scene- or video-disjoint split? This would help determine how much of the internal improvement transfers to unseen visual conditions.
  2. Can the authors provide a matched-budget baseline trained only on SpaceR/SpatialCorpus using the same base model? Comparing it with the mixed model would help identify the additional contribution of HumanPhys.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.