- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PhysCot, a benchmark of 400 physics-grounded visual sequence-completion problems for multimodal large language models. It includes three protocols: visual state perception with reduced temporal context, physical modeling with textual candidate states, and full visual reasoning. It also evaluates 24 MLLMs, and the strongest model reaches 78.56% in the mixed text-augmented condition. The benchmark aims to investigate how well MLLMs perceive physical states and reason about their evolution from visual sequences.
Critical strengths
- Predicting subsequent physical states from visual sequences is a meaningful task for evaluating the physical reasoning capabilities of MLLMs.
- The benchmark covers eight physics subfields, providing a broader evaluation scope than a benchmark focused on a single type of physical phenomenon.
- The three protocols examine model performance under different input settings, helping investigate how visual information and textual candidate states affect evaluation results.
Critical weaknesses
-
Limited dataset size and insufficiently detailed task distribution. The benchmark contains only 400 questions across eight physics subfields. However, the paper does not report the distribution of finer-grained physical scenarios within each subfield. For example, kinematics may include spring-driven motion, free fall, and motion on an inclined plane. If these scenarios are unevenly represented, the benchmark's overall results may be biased toward more frequent scenario types. The authors should provide a more detailed breakdown of the dataset by physical scenario and discuss how representative and balanced the benchmark is.
-
Potential bias introduced by strict answer parsing. Line 1186 states that "responses that fail to produce a parseable letter are scored as incorrect." This raises a concern about the validity of the evaluation protocol: a model may correctly identify the physical outcome but fail to follow the required answer format, resulting in an incorrect score. Such a failure does not necessarily indicate poor physical reasoning. The authors should report the frequency of parsing failures and consider a sensitivity analysis using a more robust answer-extraction method.
Questions for authors
- Insufficient discussion of related work. ChronoPhyBench is a relevant work that should be discussed and compared with this paper. Both benchmarks appear to involve reasoning about physical processes from sequences of images and predicting subsequent states. The authors should discuss the relationship between the two benchmarks and clarify the specific contributions and novelty of PhysCot relative to ChronoPhyBench.
- Uncertainty about data quality and physical consistency. In Figure 8, the plank moves only a short distance between Frames 2 and 3, while the correct option appears to show a much larger displacement. It is unclear whether the simulated motion is physically consistent. The authors should provide the simulation parameters and trajectory data to verify the results.
- Insufficient details on question generation and physical validation. The paper does not clearly explain how the questions and instructions are generated or how the initial physical parameters are selected. Figure 8 also lacks numerical details to verify whether the simulated motion follows physical laws. The authors should clarify these processes and explain how the physical validity of the generated data is ensured.
- Insufficient visual information in some frames. The four frames may not provide sufficient information to uniquely determine the physical outcome. For example, in Scene 1 of the supplementary material, the direction of the balls' motion is difficult to infer from the images alone. The authors should clarify whether the correct answer can be uniquely determined from the provided visual information.
19 comments
W1. Limited dataset size and insufficient detail on task distribution
(iii) Release of data, evaluation scripts, and item-level records
So that all of the above can be verified independently, we provide an anonymized data repository: https://huggingface.co/datasets/huggingface-anonymous/PhysCot
The repository contains:
- the complete 400-item dataset (with each item's initial-state parameters, four context-frame paths, four candidate paths, and the correct-answer flag);
- all 3,600 rendered images (1,600 context frames, 1,600 candidate images, 400 target frames);
- the evaluation scripts for the three protocols (
eval_reasoning.py,eval_perception.py,eval_modeling.py).
We hope the repository will allow the reviewer to check every statement above independently.
W2. Potential bias introduced by strict answer parsing
We agree. We classified every non-{A,B,C,D} response in the retained logs by cause:
| Model / protocol | Unparsed | Timeout | Auth/HTTP error | Genuine format failure (text present) |
|---|---|---|---|---|
| Kimi-K2.5 modeling | 50 | 50 | 0 | 0 |
| Kimi-K2.5 perception | 86 | 25 | 61 | 0 |
| Kimi-K2.5 reasoning | 56 | 56 | 0 | 0 |
| Opus-4.6-thinking modeling | 35 | — | — | 0 (empty response, UNKNOWN) |
| Opus-4.6-thinking perception | 12 | 0 | 12 | 0 |
| Opus-4.6-thinking reasoning | 14 | — | — | 4 |
| GPT-5.4 modeling / perception | 1 / 1 | 0 | 1 | 0 |
| Gemma-4-31B modeling / reasoning | 1 / 1 | — | 1 | 1 |
| All other files | 0 | 0 | 0 | 0 |
A dash means the cause was not broken down for that file. Those records — Opus-4.6-thinking modeling (35) and reasoning (10), and one of the two GPT-5.4 records — have empty response text, so the cause columns plus these records account for all 257 unparsed responses (131 timeouts, 75 HTTP errors, 46 empty responses without a further breakdown, and 5 text responses with no parseable letter).
From the model logs we can see the following:
- The dominant cause is transport failure, not non-compliant formatting. Of the 257 unparsed responses, 252 are API timeouts (Kimi), HTTP 401/400/524 errors, or empty responses (Opus, GPT-5.4). Only five are cases where a model produced text but no parseable letter. The situation the reviewer posits — a model that reasons correctly but formats its answer wrongly — is therefore very rare in the retained logs.
- A timeout should itself count as the model failing to solve the item. Our evaluation protocol imposes a time limit on each question. If a model does not return a parseable answer within that limit, it has not solved the physics problem within a valid reasoning budget; in evaluation semantics this is the same outcome as answering incorrectly, so we score it as an error rather than treating it as a "formatting artifact" or as noise to be excluded. On this accounting, the failures for Kimi and Opus reflect their actual behaviour on long multi-image reasoning, not incidental fluctuation in the evaluation infrastructure.
The appendix now states all three conventions. The paragraph "Answer extraction and scoring" in Appendix A.5 gives the parsing rule: the parser first looks for an explicit ANSWER: X tag and, failing that, takes the first standalone A–D letter in the response; if neither is present, the response is scored as incorrect. The same appendix states the aggregation rule: the values in the main tables average three fixed evaluation passes within each subfield, repeats are consistency checks rather than independent trials, and each retained single-pass file holds one pass of one model under one protocol, which is why the two can differ. Appendix A.5 also lists the non-response rate per model and per protocol, so readers can see directly how likely each model is to fail to complete an item within the time limit.
It is worth noting that these failures do not stem from overly strict parsing. Of the 257 unparsed responses, 252 are API timeouts, HTTP errors, or empty responses — no response text was produced to parse — so there is no case of "answered correctly but formatted improperly"; the remaining five contain text but do not specify a unique option. Even with a more permissive parser, none of the 252 records would yield an answer.
We therefore state our position plainly: when a model times out without responding, it has failed to complete the reasoning for that item within the effective time, and we count the item as one the model cannot correctly solve.
W1. Limited dataset size and insufficient detail on task distribution
We thank the reviewer for raising both points. We address scale first, then give the fine-grained scenario composition.
(i) On scale
400 questions do not mean 400 evaluation units. Each PhysCot item contains four context frames plus four candidate images — eight high-quality physics images per question — for a total of 3,200 images across the 400 items (plus 400 target frames, i.e. 3,600 rendered images in all). In terms of visual information density, PhysCot is therefore substantially larger than the item count alone suggests: it is not measured on the same scale as a "one image, one question" VQA benchmark.
A high-quality diagnostic benchmark does not need to be large. Several representative MLLM benchmarks are effective at comparable scale — MM-Vet (517 samples) and MangaVQA (526 samples) both fall in this range — and Redundancy Principles for MLLMs Benchmarks argues that for diagnostic capability, controlling instance redundancy matters more than simply increasing the number of examples. PhysCot is intended as a controlled diagnostic benchmark, not a broad-coverage evaluation suite; our goal is to provide sufficient diagnostic resolution within each physics family rather than to maximize item count.
The cost of constructing each image both limits scalability and guarantees quality. Every image requires precise control of the physical state and object attributes so that it is scientifically correct, temporally self-consistent, and visually legible. This means that only models able to understand fine-grained physical semantics and model the underlying process can solve an item — and it also means that scaling up cannot be achieved by sampling cheap data.
(ii) On task distribution
The main text reports the eight-subfield distribution (revision Figure 3); Appendix A.2 (Scenario Composition) resolves each subfield into the physical scenario families recovered from the initial-state schema and gives the difficulty composition of every family (Table 7). For direct checking, its content is reproduced below.
Scenario balance. Listing the four Combined Field variants separately describes the physical configurations; merged by physical system, the set comprises 14 families, of which twelve contain 25 items and two contain 50 (the kinematics spring–slope family and the Lorentz family). The largest family therefore accounts for 12.5% of the dataset and every other family for about 6.25%, so no single physical scenario dominates the aggregate score.
Difficulty balance. The dataset contains 130 Easy, 135 Medium, and 135 Hard items, so the three levels are almost equal in size. Difficulty is nevertheless not uniform across families: the four Kinematics families other than spring–slope, together with the Dynamics block-on-plank and the bullet–block–plank families, are 68–84% Hard, whereas thermodynamics, electrostatics, magnetic fields, combined fields, electromagnetic induction, and the work–energy gravity/spring family contain no Hard items. A cross-subfield difficulty comparison therefore partly compares families rather than difficulty levels, and the Easy$\rightarrow$Hard trend is best read as a descriptive property of this dataset rather than as a benchmark-wide guarantee.
Answer-position balance. Exactly 100 items have each of A, B, C, and D as the correct answer, so no guessing bias arises from an uneven distribution of answer positions.
Kinematics subdivisions. The five Kinematics families cover the distinct motions of this subfield: spring-driven motion on an inclined plane (spring–slope, 50 items), three-body spring motion (25), harmonic oscillation (25), motion on a curved track (25), and two-body collision (25). Free fall appears in the Work & Energy gravity/spring family.
This material is in Appendix A.2 (Table 7).