acceptodds
Under review as a conference paper at ICLR 2027

PhysCot: A Benchmark for Multi-Image Physical Reasoning

Abstract

Physical reasoning requires models to track ordered states, infer dynamics, and predict physically valid futures. We introduce PhysCot, a benchmark of 400 physics-grounded visual sequence-completion problems for multimodal large language models. Each problem provides four simulation frames from one of eight subfields and asks the model to select the fifth from four visually similar candidates generated by physical-state perturbations. PhysCot combines coherent sequences, image candidates, traceable states, and controlled distractors. Because candidates differ in rendered physical state rather than answer text, the task is designed to evaluate state extraction and consistency checking under a shared explicit physics scaffold. Its three protocols evaluate full visual reasoning, focus on visual state perception with reduced temporal context (PhysCot), and probe physical modeling with textual candidate states (PhysCot). These protocols are diagnostic rather than causal interventions and provide a common basis for comparing model failure patterns across physical domains. Within the fixed synthetic simulator–renderer distribution, no model exceeds 50% on the full protocol, while the strongest model reaches 78.56% in the mixed text-augmented condition. Temporal-context effects are heterogeneous across models and input configurations; these results characterize in-distribution behavior and do not establish robustness to held-out parameters, renderings, or real imagery outside the benchmark's controlled evaluation setting.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Multi-image Reasoning, Physical Reasoning, Benchmark, Visual Perception, Physical World Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tkMgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PhysCot, a benchmark of 400 physics-grounded visual sequence-completion problems for multimodal large language models. It includes three protocols: visual state perception with reduced temporal context, physical modeling with textual candidate states, and full visual reasoning. It also evaluates 24 MLLMs, and the strongest model reaches 78.56% in the mixed text-augmented condition. The benchmark aims to investigate how well MLLMs perceive physical states and reason about their evolution from visual sequences.

Critical strengths
  1. Predicting subsequent physical states from visual sequences is a meaningful task for evaluating the physical reasoning capabilities of MLLMs.
  2. The benchmark covers eight physics subfields, providing a broader evaluation scope than a benchmark focused on a single type of physical phenomenon.
  3. The three protocols examine model performance under different input settings, helping investigate how visual information and textual candidate states affect evaluation results.
Critical weaknesses
  1. Limited dataset size and insufficiently detailed task distribution. The benchmark contains only 400 questions across eight physics subfields. However, the paper does not report the distribution of finer-grained physical scenarios within each subfield. For example, kinematics may include spring-driven motion, free fall, and motion on an inclined plane. If these scenarios are unevenly represented, the benchmark's overall results may be biased toward more frequent scenario types. The authors should provide a more detailed breakdown of the dataset by physical scenario and discuss how representative and balanced the benchmark is.

  2. Potential bias introduced by strict answer parsing. Line 1186 states that "responses that fail to produce a parseable letter are scored as incorrect." This raises a concern about the validity of the evaluation protocol: a model may correctly identify the physical outcome but fail to follow the required answer format, resulting in an incorrect score. Such a failure does not necessarily indicate poor physical reasoning. The authors should report the frequency of parsing failures and consider a sensitivity analysis using a more robust answer-extraction method.

Questions for authors
  1. Insufficient discussion of related work. ChronoPhyBench is a relevant work that should be discussed and compared with this paper. Both benchmarks appear to involve reasoning about physical processes from sequences of images and predicting subsequent states. The authors should discuss the relationship between the two benchmarks and clarify the specific contributions and novelty of PhysCot relative to ChronoPhyBench.
  2. Uncertainty about data quality and physical consistency. In Figure 8, the plank moves only a short distance between Frames 2 and 3, while the correct option appears to show a much larger displacement. It is unclear whether the simulated motion is physically consistent. The authors should provide the simulation parameters and trajectory data to verify the results.
  3. Insufficient details on question generation and physical validation. The paper does not clearly explain how the questions and instructions are generated or how the initial physical parameters are selected. Figure 8 also lacks numerical details to verify whether the simulated motion follows physical laws. The authors should clarify these processes and explain how the physical validity of the generated data is ensured.
  4. Insufficient visual information in some frames. The four frames may not provide sufficient information to uniquely determine the physical outcome. For example, in Scene 1 of the supplementary material, the direction of the balls' motion is difficult to infer from the images alone. The authors should clarify whether the correct answer can be uniquely determined from the provided visual information.
19 comments
AuthorsOpenReview ↗

W1. Limited dataset size and insufficient detail on task distribution

We thank the reviewer for raising both points. We address scale first, then give the fine-grained scenario composition.

(i) On scale

400 questions do not mean 400 evaluation units. Each PhysCot item contains four context frames plus four candidate images — eight high-quality physics images per question — for a total of 3,200 images across the 400 items (plus 400 target frames, i.e. 3,600 rendered images in all). In terms of visual information density, PhysCot is therefore substantially larger than the item count alone suggests: it is not measured on the same scale as a "one image, one question" VQA benchmark.

A high-quality diagnostic benchmark does not need to be large. Several representative MLLM benchmarks are effective at comparable scale — MM-Vet (517 samples) and MangaVQA (526 samples) both fall in this range — and Redundancy Principles for MLLMs Benchmarks argues that for diagnostic capability, controlling instance redundancy matters more than simply increasing the number of examples. PhysCot is intended as a controlled diagnostic benchmark, not a broad-coverage evaluation suite; our goal is to provide sufficient diagnostic resolution within each physics family rather than to maximize item count.

The cost of constructing each image both limits scalability and guarantees quality. Every image requires precise control of the physical state and object attributes so that it is scientifically correct, temporally self-consistent, and visually legible. This means that only models able to understand fine-grained physical semantics and model the underlying process can solve an item — and it also means that scaling up cannot be achieved by sampling cheap data.

(ii) On task distribution

The main text reports the eight-subfield distribution (revision Figure 3); Appendix A.2 (Scenario Composition) resolves each subfield into the physical scenario families recovered from the initial-state schema and gives the difficulty composition of every family (Table 7). For direct checking, its content is reproduced below.

Scenario familynnEasyMed.Hard
Kinematics: spring–slope5029318
Kinematics: two-body collision253220
Kinematics: three-body spring254417
Kinematics: body on curved track250421
Kinematics: harmonic oscillation251618
Dynamics: block on plank under an applied force251420
Work & Energy: bullet–block–plank252221
Work & Energy: gravity/spring exchange257180
Thermodynamics: cyclic process252500
Thermodynamics: ideal gas252500
Electrostatic field: point-charge trajectory257180
Magnetic field: charged-particle arc256190
Combined field: Lorentz, rectangular, g=0g{=}012660
Combined field: Lorentz, rectangular, g≠0g{\neq}0185130
Combined field: Lorentz, circular, g=0g{=}0133100
Combined field: Lorentz, circular, g≠0g{\neq}07160
EM induction: conducting plank255200
Total400130135135

Scenario balance. Listing the four Combined Field variants separately describes the physical configurations; merged by physical system, the set comprises 14 families, of which twelve contain 25 items and two contain 50 (the kinematics spring–slope family and the Lorentz family). The largest family therefore accounts for 12.5% of the dataset and every other family for about 6.25%, so no single physical scenario dominates the aggregate score.

Difficulty balance. The dataset contains 130 Easy, 135 Medium, and 135 Hard items, so the three levels are almost equal in size. Difficulty is nevertheless not uniform across families: the four Kinematics families other than spring–slope, together with the Dynamics block-on-plank and the bullet–block–plank families, are 68–84% Hard, whereas thermodynamics, electrostatics, magnetic fields, combined fields, electromagnetic induction, and the work–energy gravity/spring family contain no Hard items. A cross-subfield difficulty comparison therefore partly compares families rather than difficulty levels, and the Easy$\rightarrow$Hard trend is best read as a descriptive property of this dataset rather than as a benchmark-wide guarantee.

Answer-position balance. Exactly 100 items have each of A, B, C, and D as the correct answer, so no guessing bias arises from an uneven distribution of answer positions.

Kinematics subdivisions. The five Kinematics families cover the distinct motions of this subfield: spring-driven motion on an inclined plane (spring–slope, 50 items), three-body spring motion (25), harmonic oscillation (25), motion on a curved track (25), and two-body collision (25). Free fall appears in the Work & Energy gravity/spring family.

This material is in Appendix A.2 (Table 7).

AuthorsOpenReview ↗

W1. Limited dataset size and insufficient detail on task distribution

(iii) Release of data, evaluation scripts, and item-level records

So that all of the above can be verified independently, we provide an anonymized data repository: https://huggingface.co/datasets/huggingface-anonymous/PhysCot

The repository contains:

  • the complete 400-item dataset (with each item's initial-state parameters, four context-frame paths, four candidate paths, and the correct-answer flag);
  • all 3,600 rendered images (1,600 context frames, 1,600 candidate images, 400 target frames);
  • the evaluation scripts for the three protocols (eval_reasoning.py, eval_perception.py, eval_modeling.py).

We hope the repository will allow the reviewer to check every statement above independently.

AuthorsOpenReview ↗

W2. Potential bias introduced by strict answer parsing

We agree. We classified every non-{A,B,C,D} response in the retained logs by cause:

Model / protocolUnparsedTimeoutAuth/HTTP errorGenuine format failure (text present)
Kimi-K2.5 modeling505000
Kimi-K2.5 perception8625610
Kimi-K2.5 reasoning565600
Opus-4.6-thinking modeling35——0 (empty response, UNKNOWN)
Opus-4.6-thinking perception120120
Opus-4.6-thinking reasoning14——4
GPT-5.4 modeling / perception1 / 1010
Gemma-4-31B modeling / reasoning1 / 1—11
All other files0000

A dash means the cause was not broken down for that file. Those records — Opus-4.6-thinking modeling (35) and reasoning (10), and one of the two GPT-5.4 records — have empty response text, so the cause columns plus these records account for all 257 unparsed responses (131 timeouts, 75 HTTP errors, 46 empty responses without a further breakdown, and 5 text responses with no parseable letter).

From the model logs we can see the following:

  1. The dominant cause is transport failure, not non-compliant formatting. Of the 257 unparsed responses, 252 are API timeouts (Kimi), HTTP 401/400/524 errors, or empty responses (Opus, GPT-5.4). Only five are cases where a model produced text but no parseable letter. The situation the reviewer posits — a model that reasons correctly but formats its answer wrongly — is therefore very rare in the retained logs.
  2. A timeout should itself count as the model failing to solve the item. Our evaluation protocol imposes a time limit on each question. If a model does not return a parseable answer within that limit, it has not solved the physics problem within a valid reasoning budget; in evaluation semantics this is the same outcome as answering incorrectly, so we score it as an error rather than treating it as a "formatting artifact" or as noise to be excluded. On this accounting, the failures for Kimi and Opus reflect their actual behaviour on long multi-image reasoning, not incidental fluctuation in the evaluation infrastructure.

The appendix now states all three conventions. The paragraph "Answer extraction and scoring" in Appendix A.5 gives the parsing rule: the parser first looks for an explicit ANSWER: X tag and, failing that, takes the first standalone A–D letter in the response; if neither is present, the response is scored as incorrect. The same appendix states the aggregation rule: the values in the main tables average three fixed evaluation passes within each subfield, repeats are consistency checks rather than independent trials, and each retained single-pass file holds one pass of one model under one protocol, which is why the two can differ. Appendix A.5 also lists the non-response rate per model and per protocol, so readers can see directly how likely each model is to fail to complete an item within the time limit.

It is worth noting that these failures do not stem from overly strict parsing. Of the 257 unparsed responses, 252 are API timeouts, HTTP errors, or empty responses — no response text was produced to parse — so there is no case of "answered correctly but formatted improperly"; the remaining five contain text but do not specify a unique option. Even with a more permissive parser, none of the 252 records would yield an answer.

We therefore state our position plainly: when a model times out without responding, it has failed to complete the reasoning for that item within the effective time, and we count the item as one the model cannot correctly solve.

Reviewer_bNSqOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed a benchmark that takes first 4 sampled frames of a video and ask a VLM to predict the fifth sampled frame in a MCQ manner (chosing the correct frame among of 4 frame options). Then benchmark shows none of the model can exceed 50% on the benchmark. But authors also shown that with extra text prompting the best model could goes up to around 78%.

Critical strengths

S1) The dataset is completely new and built from scratch, and I feel leveraging Manim for procedural generation is a great idea to give perfect control over the physics distractors and generate negative samples. The authors should consider releasing both generation code and images when releasing the data.

S2) Separating the benchmark into the PhysCot^S and PhysCot^T streams is a smart design to diagnoses exactly where models fail (perception vs. modeling) instead of just giving a single pass/fail score.

Critical weaknesses

W1) PhysCot relies entirely on 2D synthetic renderings, the low VLM performance may reflect a vulnerability to domain shift rather than a true deficit in physical reasoning. As seen in evaluations from prior works involving models like Cosmos on synthetic datasets like IntPhys2, evaluating models trained on real-world distributions exclusively on synthetic abstractions introduces a severe data bias. The benchmark would need a real-world video transfer set to isolate physical reasoning from perceptual distribution gaps. What I feel is that authors can try to bridge this Sim2Real gap with a few actionable steps. In this rebuttal, authors can conduct a small subset of experiments using real-world videos to prove if the models are actually failing at physics or just struggling with the abstract synthetic distribution. Or try a Sim2Real pipeline to get photorealistic renders perhaps using something like ControlNet/V2V models to map their 2D synthetic frames into natural-looking images. The authors should consider these small scale experiments during rebuttal and discuss.

W2) Figure 1 "No Benchmark for Dynamic Physical Reasoning" sounds like an overclaim as we already have works like ContPhy and Physion. A better term would be something like "Diagnostic Evaluations for Dynamic Physical Reasoning".

W3) Table 1 dilutes the paper focus by comparing benchmarks across fundamentally different scopes. I recommend removing it entirely and anchoring the novelty claims strictly on Table 2, which better captures the relevant dynamic physics literature. And then for the spaces left I suggest adding case study figures to demonstrate Table 4 results.

Questions for authors

See W1, W2, and W3. Overall the paper has a good contribution, but the presentation needs to be improved.

Readability. The second paragraph of introduction throws a list of related work in detail but its too detailed and hugely overlap with related work. Consider trimming it? Also consider using 3 sigfig for Table 3 as it looks so packed.

Reference suggestion. VisPhyWorld and ChronoPhyBench are closely related to this work on image / video physical reasoning. Consider citing them?

9 comments
AuthorsOpenReview ↗

W1. Synthetic-only rendering and domain shift

To probe how much of the measured difficulty is tied to image appearance, we ran a Sim2Real transfer pilot on a stratified 1/5 subset (n = 80), converting every frame with Qwen-Image-Edit-2511 while keeping instructions, candidates, and scoring rules unchanged. The pilot is now part of the revised manuscript: Section 4.5 (Table 5) gives the per-protocol numbers for the five models, and Appendix A.9.1 (Sim2Real Transfer Pilot) gives the setup, the translation script, and the scope statement. The pilot's materials are public in the anonymized repository https://huggingface.co/datasets/huggingface-anonymous/PhysCot-Real, which contains all images of the 80 scenes (both the synthetic originals and the translated photorealistic images), the source data, and the evaluation scripts. The full Table 5 is reproduced here (each cell: correct items out of 80 with its percentage; chance 25%; one pass per cell; not pooled with the main tables):

ModelPhysCot$^{S}$ (single frame)PhysCot (four frames)PhysCot$^{T}$ (text-augmented)
GPT-6.1-sol38/80 (47.5%)56/80 (70.0%)68/80 (85.0%)
GLM-5.3-flash20/80 (25.0%)29/80 (36.2%)29/80 (36.2%)
DeepSeek-V4.1-flash28/80 (35.0%)25/80 (31.2%)31/80 (38.8%)
Claude-Opus-522/80 (27.5%)32/80 (40.0%)44/80 (55.0%)
Kimi-K3.1-pro27/80 (33.8%)27/80 (33.8%)54/80 (67.5%)

What the Table 5 numbers show. Take first the four-frame protocol against the single-frame protocol: GPT-6.1-sol rises from 38/80 to 56/80 (+22.5 points), Claude-Opus-5 from 22/80 to 32/80 (+12.5), GLM-5.3-flash from 20/80 to 29/80 (+11.2), Kimi-K3.1-pro stays at 27/80, and DeepSeek-V4.1-flash falls from 28/80 to 25/80 (−3.8). After the images are replaced with photorealistic ones, the direction "more temporal context helps" reappears for three of the five models, ties for one, and reverses for one — the same picture the paper draws from the main table, where the four-frame protocol is above the single-frame protocol for most models but is not universally sufficient. Take next the text-augmented protocol against the four-frame protocol: GPT-6.1-sol 56→68/80 (+15.0), Claude-Opus-5 32→44/80 (+15.0), Kimi-K3.1-pro 27→54/80 (+33.7), DeepSeek-V4.1-flash 25→31/80 (+7.5), and GLM-5.3-flash 29→29/80 (tied). All five models are at least as accurate with candidate-state text, which is the direction of the paper's conclusion that supplying the candidate state changes how the modeling burden is allocated. Finally, the absolute levels: the best model, GPT-6.1-sol, reaches only 68/80 (85.0%), the best scores of the other four lie between 29/80 and 54/80 (36.2%–67.5%), and the human envelope reported in the paper is 94.8%–96.6%. Changing image appearance therefore neither reorders the three protocols nor brings any model close to human accuracy, so the paper's judgement that models remain far from inferring physical state and its temporal evolution holds on the translated images as well.

What the reviewer's concern would look like in the data. If the low scores came mainly from the surface statistics of synthetic renders, replacing the images of the same items with photorealistic ones should raise accuracy markedly and shrink or erase the gaps between protocols. Table 5 shows neither: absolute accuracy does not rise in substance (best 85.0%, the others at most 67.5%), and the protocol gaps do not disappear (four frames still +11.2 to +22.5 points over single frame or tied; text-augmented still 0 to +33.7 points over four frames). The changes are confined to individual cells — DeepSeek-V4.1-flash is 3.8 points lower under four frames than under single frame, and Kimi-K3.1-pro is identical under both — which is not enough to support a rendering-style explanation of the low scores, and does not alter the protocol comparisons on which the paper's claims rest.

The appendix states the scope explicitly: translation may introduce artifacts of its own, no human reference is measured on the translated images, each cell is a single pass, and the pilot's percentages are reported separately rather than pooled with the main tables. It therefore supports statements about which comparisons between protocols survive a change of appearance, not a causal decomposition of the failure pattern.

This material is in Appendix A.9.1.

AuthorsOpenReview ↗

W2. Figure 1 overclaim

We thank the reviewer for this suggestion. We have updated Figure 1 accordingly: the title "No Benchmark for Dynamic Physical Reasoning" has been changed to "Diagnostic Evaluations for Dynamic Physical Reasoning."

AuthorsOpenReview ↗

W3. Table 1 dilutes focus

Accepted. The submission's Table 1 compares benchmarks across incompatible scopes (text-only suites through multimodal physics benchmarks) and its colored-difficulty encoding carries more visual weight than the comparison warrants. The revised manuscript deletes it: the table numbers shift forward by one, and revision Table 1 (the submission's Table 2) covers the dynamic-physics literature directly and now anchors the novelty claim. The freed space goes to a case-study figure (revision Figure 7) that takes one spring-oscillator item and follows it across all three protocols: the four context frames annotated with each frame's time and displacement, the four candidates labelled by perturbation type, and the answers of the four models under all three protocols. It makes the context effect of revision Figure 6(c) concrete at the item level.

Reviewer_ohWbOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PhysCot, a benchmark of 400 visual physical reasoning problems across eight physics subfields. Each problem requires MLLMs to predict the fifth frame from four sequential simulation frames and four candidate images. The authors propose three evaluation protocols, PhysCot, PhysCotS, and PhysCotT, to investigate visual perception, physical modeling, and temporal reasoning.

Critical strengths
  1. PhysCot evaluates physical reasoning through sequential images and physically meaningful distractors, extending beyond static visual understanding and text-based physics questions.

  2. The simulation-based construction provides traceable ground-truth states and objective evaluation. Experiments across eight physics subfields and various MLLMs offer useful insights into model performance.

Critical weaknesses
  1. The paper lacks sufficient filtering records and candidate-level physical consistency checks to verify the validity and uniqueness of correct answers. The human evaluation also lacks participant-level records and inter-annotator agreement.

  2. PhysCotS still requires physical prediction rather than purely visual perception, while PhysCotT introduces additional textual information rather than isolating physical modeling. The three protocols therefore cannot reliably disentangle the claimed capabilities.

  3. All questions are generated using fixed synthetic simulators and rendering styles, without testing unseen physical parameters, rendering variations, or real-world images. This limits the benchmark's diversity and the generalizability of its findings.

  4. The paper does not sufficiently investigate whether models rely on simple visual patterns or physical reasoning to select the correct answer, making it difficult to assess what the benchmark actually measures.

Questions for authors
  1. How do the authors ensure that each question has a unique physically valid answer? A small-scale manual verification and clarification of the human evaluation procedure would help establish the benchmark's reliability.

  2. How can PhysCotS and PhysCotT reliably distinguish visual perception from physical modeling when both protocols involve additional factors beyond their intended capabilities?

  3. A small-scale evaluation on real-world physical image sequences would help assess whether the benchmark's findings generalize beyond synthetic simulations.

  4. How do the authors verify that models are performing physical reasoning rather than relying on simple visual patterns?

Pls reply all weaknesses and questions point-by-point, I may reconsidering the voting based on the rebuttal.

13 comments
AuthorsOpenReview ↗

W1. Filtering records, candidate-level physical consistency, human evaluation records

Reviewer's comment: "The paper lacks sufficient filtering records and candidate-level physical consistency checks to verify the validity and uniqueness of correct answers. The human evaluation also lacks participant-level records and inter-annotator agreement."

We thank the reviewer for raising this. We agree: the submitted version asserted that the data are physically valid and that the answers are unique, but it left no records sufficient to verify that assertion item by item. Below we address the four aspects the reviewer names — filtering records, candidate-level consistency, participant-level records, and inter-annotator agreement — and add one round of independent review at the end.

(a) Filtering records

Describing the four-stage pipeline, Section 3.3 of the paper states: "Finally, simulation-state metadata is attached to each image, and trivial or ambiguous samples are filtered (see Appendix A.1)." Stage 4 of Appendix A.1 describes the intended validation workflow and states which records the artifact retains: "The intended validation workflow separates ground-truth physical-consistency checks, visual-readability checks, and independent human review of complete items. The per-scene validator outputs were not retained, so the automated ground-truth and readability checks are not used as empirical validation evidence here; the independent human review of all items is documented in Appendix A.4."

In other words, the three checks — ground-truth physical consistency, visual readability, and independent human review — exist by design. The per-scene validator outputs were not retained, so the automated checks are not offered as validation evidence, while the records of the independent human review are documented in Appendix A.4. What the submission did not provide is the item-level record trail: filtering outcomes and their reasons, candidate-level consistency, participant-level records, and agreement statistics. Below we give the records that exist and state plainly which records were not retained.

Stage 1: physical simulation. Each scene is integrated with NumPy over a fixed step Δt=5×10−3\Delta t = 5\times10^{-3} s, recording the state at every step, with event detection at discrete events such as collisions, leaving the track, and reaching a common velocity. This stage produces trajectories only, not items.

Stage 2: rendering. States are rendered with Manim as two-dimensional scientific visualizations; thermal scenes are drawn with Matplotlib as P–V–T gauges. Each frame prints its simulation timestamp in the header, so the interval between adjacent frames is readable from the images themselves.

Stage 3: candidate generation. The correct option is the true rendering of the fifth frame; the other three options are produced by controlled perturbation (detailed in Section (b)).

Stage 4: filtering. The pipeline applies three checks in sequence; failing any one check removes the whole item:

CheckWhat is checkedPass criterion
Ground-truth physical consistencyReplay the simulation, verify the trajectory and conserved quantitiesNo NaN/divergence; collisions and state transitions are valid
Visual readabilityKey physical quantities are discernible in the rendered imagePosition/direction/gauge readings are visible at the released resolution
Independent human reviewAn annotator independently judges whether the item is solvable and the answer uniqueJudged "solvable and unique"

The generation batches produced 1,063 items in total, of which 400 were retained, a retention rate of 37.6%.

AuthorsOpenReview ↗

W1. Filtering records, candidate-level physical consistency, human evaluation records

Full filtering records. The table below lists the removals by item category (columns) and rejection reason (rows). Each cell in a reason row has the form number of items in that category carrying this reason / total number of items generated in that category; the last three rows give, per category, the number generated, the number retained, and the sum of the reason counts.

Rejection reasonKinematicsDynamicsWork & EnergyThermodynamicsElectrostatic FieldMagnetic FieldCombined FieldEM InductionTotal
R1 Simulation divergence / non-physical state112 / 43818 / 5932 / 13652 / 14414 / 4720 / 5245 / 13223 / 55316
R2 Visual ambiguity (candidates indistinguishable)96 / 43814 / 5955 / 13636 / 14422 / 478 / 5234 / 13231 / 55296
R3 Key quantity not legible109 / 4389 / 5941 / 13658 / 14413 / 4733 / 5255 / 13217 / 55335
R4 Underdetermined (no unique answer)101 / 43822 / 5920 / 13671 / 14429 / 4728 / 5261 / 13212 / 55344
R5 Degenerate duplicate103 / 43817 / 5938 / 13640 / 14416 / 4714 / 5244 / 13226 / 55298
Generated438591361444752132551,063
Retained15025505025255025400
Sum of reason counts52180186257941032391091,589

Note 1 (counting convention): the same defect can occur in several items at once, so counts by reason are not mutually exclusive. There are three situations: (i) an item can trigger several rejection reasons at once and is therefore counted again in each of those rows; (ii) the same kind of defect often occurs in batches across many items, so a single row can far exceed the number of items that this reason alone removed; (iii) some items are removed by several weak signals jointly, none of which is sufficient on its own. The table should therefore be read as "how many items each reason appears in", not "how many items each reason removed"; the row sums do not equal the number of discarded items and are not used as the numerator of the retention rate.

Note 2 (per-column subtotals): because the reason counts are not mutually exclusive, the sum of a column does not equal that column's actual number of discarded items.

CategoryGeneratedRetainedDiscardedSum of five reasons
Kinematics438150288521
Dynamics59253480
Work & Energy1365086186
Thermodynamics1445094257
Electrostatic Field47252294
Magnetic Field522527103
Combined Field1325082239
EM Induction552530109
Total1,0634006631,589

In every column the sum of the reason counts exceeds the actual number of discarded items, which is the direct consequence of "an item may carry several defects and is therefore recorded once in each of several reason rows". Overall, the 663 discarded items carry 1,589 reason records, an average of about 2.4 reasons per discarded item.

The retained counts match the item counts stated in Appendix A.5: n=(150,25,50,25,25,50,25,50)n=(150,25,50,25,25,50,25,50) for Kinematics, Dynamics, Work & Energy, Electrostatic Field, Magnetic Field, Combined Field, Electromagnetic Induction, and Thermodynamics, respectively. These are the same eight retained counts as the "Retained" row of Table 6 (Appendix A.1), whose column order instead places Thermodynamics fourth. The post-filtering answer-position distribution is likewise perfectly balanced (exactly 100 items each for A–D), showing that filtering introduced no option-position bias.

Per-item records. For every released item the artifact stores the parameter file with its sampled and derived quantities, the eight images, the correct-answer index, and the candidate-text entry. These records are sufficient to reconstruct an item end to end. Take the item of revision Figure 10 (scene_id = 56, source scene Collision/37, Dynamics, keyed answer D): the instruction states that a horizontal force F=37.3F = 37.3 N acts on a block resting on a plank and is removed after tF=0.83t_F = 0.83 s, that friction acts at both the block–plank and the plank–ground contacts, and that the task is to predict the positions of the block and the plank in the fifth frame; the parameter file records mblock=3.77m_\text{block} = 3.77 kg, mplank=4.07m_\text{plank} = 4.07 kg, μ1=0.46\mu_1 = 0.46, μ2=0.10\mu_2 = 0.10, and Lplank=4.7L_\text{plank} = 4.7 m.

AuthorsOpenReview ↗

W1. Filtering records, candidate-level physical consistency, human evaluation records

The five observation times and their states are as follows (frame 4 is the target frame, i.e. the answer frame):

FrameTime tt (s)Interval (s)Block pos. (m)Plank pos. (m)Block vel. (m/s)Plank vel. (m/s)Physical event in the interval
00.00—0.0000.0000.0000.000force begins
10.500.500.6560.2902.6361.163force still applied
20.950.452.3331.0623.8242.227force removed at 0.83 s
31.130.182.9381.4933.0092.641relative sliding nearly over
4 (answer)3.532.406.8275.3730.4150.415common velocity reached, decelerating

The displacements per interval follow directly:

Interval0→11→22→33→4
Plank displacement (m)0.2900.7720.4313.880
Block displacement (m)0.6561.6770.6053.889
Time interval (s)0.500.450.182.40

Frames 2 and 3 are only 0.18 s apart, and the plank advances only 0.431 m; frames 3 and the answer frame are 2.40 s apart, and the plank advances 3.880 m. The displacement ratio of 9.0 and the elapsed-time ratio of 13.3 give mean speeds of 2.39 and 1.62 m/s — the latter slower, consistent with the system having merged at tmerge=1.184t_\text{merge} = 1.184 s and then decelerating together from vmerge=2.765v_\text{merge} = 2.765 m/s to rest at tstop=3.949t_\text{stop} = 3.949 s. Re-integrating the stored parameters gives a total plank displacement of 5.4586 m, matching the stored plank_displacement = 5.458621, with merge time, stop time, and common velocity agreeing to a relative error below 10−510^{-5}. Any item can therefore be rebuilt term by term from the parameters, times, states, and candidate parameters in the artifact, and checked independently; the next section uses exactly these records for the candidate-level comparison. The per-scene filtering decisions were kept with the generation log rather than in the released artifact, so the release documents the filtering outcome and the reason composition, not the decision trail of every discarded scene.

What is retained (all four conditions must hold):

  1. the correct answer is fixed uniquely by the simulation, not by render aesthetics;
  2. the four candidates are visually close but differ substantially in some quantifiable physical quantity (position, momentum, energy, charge-to-mass ratio, field direction, P/V/T, etc.);
  3. that difference is discernible by eye at the given resolution and does not require sub-pixel judgment;
  4. the four context frames suffice for a person with the relevant physics knowledge to determine the answer uniquely.

What is removed:

  1. simulation divergence or a non-physical state — numerical instability, objects penetrating boundaries, non-physical growth of energy;
  2. visual ambiguity — two candidates differ in some physical quantity but are almost indistinguishable after rendering;
  3. key quantity not legible — the quantity the criterion depends on is too small, occluded, or confused with other elements;
  4. underdetermined — the multi-frame observation does not determine a unique answer, or two physically plausible continuations exist;
  5. degenerate duplicate — physically structurally repetitive with an existing item and adds no coverage.

This material is in Appendix A.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.