- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CognPhys is a benchmark for testing visual physical-rule induction in procedurally generated, non-Newtonian worlds. It contains 284 MuJoCo worlds: Track A has 150 single-law worlds for parameter estimation, and Track B has 134 worlds composed from a closed vocabulary of eleven laws for rule identification. A third task evaluates future trajectory prediction. The design uses parameter-twin worlds, a same-pixel classical estimator as an identifiability check/reference, and a second camera view for Track A. Across 18 closed-source API models and five local open-weight models, the best exact-set rule-determination score is 20.1% (versus 1.6% for random subsets), the best parameter accuracy within 10% is 15.3% (versus 90.7% for a classical estimator), and the best future-prediction hit rate within 0.15 ball radii is 33.3% (versus 87% for replay with the true rules). The results provide a careful diagnosis within a controlled synthetic benchmark, with more limited support for open-ended or real-world physical law discovery.
Critical strengths
- CognPhys combines video input, multiple counterfactual physics worlds, and explicit rule/parameter tasks in a coherent benchmark. Track A and Track B separate quantitative parameter estimation from closed-vocabulary rule identification.
- The identifiability gate is a useful design choice: worlds are retained only when a classical method can recover the parameter from the same pixels. Twin worlds that differ in one parameter and the second camera view further help diagnose whether a model is responding to visual evidence.
Critical weaknesses
- The related-work coverage is incomplete at the point most relevant to novelty. The manuscript surveys fixed-physics visual benchmarks, numerical law-discovery tasks, and general physical-reasoning benchmarks, but does not cite or compare InPhyRe, a 2025 preprint accepted to TMLR in 2026 that tests inductive reasoning under unseen physics using synthetic collision videos; QuantiPhy, a CVPR 2026 video benchmark for quantitative size, velocity, and acceleration inference; or PhysBench, which evaluates physical-world understanding interleaved video-image-text entries in four domains, including physics-based dynamics. These are not identical tasks: InPhyRe focuses on predicting collision outcomes after demonstrations, QuantiPhy uses known physical relations and priors, and PhysBench covers broader physical-world understanding rather than CognPhys’s controlled non-Newtonian rule and parameter induction. They are nevertheless close enough that Section 2 and Table 1 should explain the distinctions. Their omission leaves the “first benchmark to unify” positioning and the claim that video benchmarks do not test induction insufficiently substantiated.
- The headline reference comparison is easy to misread and hides stronger simple baselines. Table 3 places 87% future-prediction accuracy under “same-pixel ceiling,” but this value comes from replaying the simulator with the ground-truth rules and parameters; it is an oracle, not a pixel-based predictor available to a model. Appendix Table 6 reports a 43% no-video last-velocity predictor, which exceeds the best VLM’s 33.3%, and a 24% no-video family-median parameter predictor, which exceeds the best VLM’s 15.3%. These baselines should appear in the main table and be clearly separated from oracle ceilings. For Track B, the 1.6% uniformly random rule-subset baseline is weak; a majority-set or per-rule base-rate predictor is needed before “well above chance” can be interpreted as evidence of useful rule induction.
- Model coverage and reported denominators are not comparable enough for the displayed rankings. Figure 7 reports 150 Track A worlds for closed-source models but only 40 for open-weight models, omitting two rule families; the main table does not give per-model denominators or a same-world comparison. Future prediction is based on only 30 worlds. In addition, Appendix A.4 reports 2,818 model-world observations for Track A’s log-error decomposition, whereas the stated coverage implies 18×150 + 5×40 = 2,900 possible observations. Since unparsable outputs are counted incorrect in the main evaluation, the paper should state whether and why 82 observations are absent from this analysis. Figure 6 reports binomial intervals without making the independent sampling unit clear; Track A twin worlds share seeds and initial conditions, and each world is evaluated by multiple models. If these observations are treated as independent, the uncertainty is understated. Please report paired/cluster-aware intervals and API decoding or repeat-call variability.
- The visual evidence and the interpretation of failure need stronger validation. Clips are 12–20 seconds in Track A and 14 seconds in Track B, while models receive frames described as 2 fps with a maximum of 16. Section 3.2 says frames are evenly spaced, but the paper does not make clear how that rule interacts with the cap for each clip length or show sensitivity to the resulting temporal sampling. More importantly, the Track A identifiability gate validates parameter recovery by a hand-written estimator, not all visual signatures in Track B. The 97.5% forced choice between counterfactual twins is a different, easier discrimination task and does not establish that perception is solved for all eleven composed rules. The conclusion that the bottleneck is induction rather than perception is therefore stronger than the diagnostic evidence supports.
- Implementation provenance and reproducibility are not documented at sufficient detail. The paper describes a procedural rule engine, MuJoCo simulation, rule templates, and state logging, but does not state how the simulator programs were authored (handwritten versus code-generation or AI coding assistance), how each rule is implemented in the engine, or what independent unit/regression tests validate the event conditions and ground-truth labels. These implementation choices define the physical phenomena being benchmarked, so authorship workflow, implementation details, and validation artifacts are important for assessing validity and reproducibility.
Questions for authors
- Can the main results table include the no-video family-median and last-velocity baselines, while labeling ground-truth simulator replay explicitly as an oracle ceiling? For Track B, can you add majority-set and per-rule prevalence baselines?
- Please report per-task and per-model sample counts, the exact 40-world open-weight subset and omitted families, and a same-world comparison against the closed-source models. How do the 2,818 observations in Appendix A.4 relate to the 2,900 expected Track A model-world evaluations?
- How are the 2 fps and 16-frame cap applied to 12–20 second clips: are frames selected across the full clip or truncated? Can you provide a sensitivity analysis showing that the evidence-bearing events remain visible under this sampling policy?
- Can the uncertainty analysis account for paired twin worlds and shared worlds across models, and report API decoding/repeat-call stability? What additional visual-evidence diagnostic supports the claim that rule induction, rather than perception, is the main bottleneck on Track B?
- How were the world-generator and simulator programs authored: handwritten by the authors, generated in part with coding assistants/LLMs (including vibe-coding-style workflows), or by another workflow? Please disclose any code-generation assistance, explain how each rule maps to MuJoCo (including timestep, force, and event handling), and provide implementation/configuration details plus unit or validation tests that reproduce the 284 worlds and their ground-truth labels.