- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SPARKBench, a human-centric spatio-physics reasoning benchmark containing 5,291 questions over 1,442 images. It covers 3D pose, orientation, relations, collision, articulation/kinematics, and intuitive physics, with both descriptive and counterfactual questions. Real photographs provide diverse scenes, while procedural and photo-derived renders provide stored geometry and physical state for answer verification. Questions undergo source-specific automatic checks followed by human review.
The paper evaluates twelve vision-language models and identifies weaknesses in orientation and kinematics. All ten open models perform worse on counterfactual than descriptive questions. Using Qwen3-VL-8B, the paper further compares Image, Graph, and Code representations under estimated and oracle states. Explicit body structure and derived relations improve some spatial judgments, while physical reasoning benefits from relevant state variables and executable checks on known synthetic state.
Critical strengths
-
The benchmark connects articulated body structure with physical constraints and provides verifiable labels for part of its data. It combines body-centered reference frames, spatial relations, contact and interpenetration, joint limits, reach, and support. Procedural scenes retain geometry and physical state, with documented pose screening, deduplication, visibility checks, decision rules, and margins. Photo-derived scenes support controlled edits while explicitly treating fitted bodies as proxies rather than measured ground truth for the source photographs.
-
The representation and execution analyses include useful negative results and diagnostic references. Estimated graphs improve some pose and orientation judgments but degrade several physical and real-image results. Operation-selection and missing-field statistics expose practical limitations of the estimated-state pipeline. Oracle comparisons demonstrate potential benefits of physical state and computation, while the authors acknowledge that state accuracy and field coverage change together. The cases also illustrate missed detections, reference-frame errors, incorrect entity binding, and failures to use available state correctly.
Critical weaknesses
-
Image answerability and annotation reliability are only partially characterized. The paper provides geometric verification, three-annotator screening, visual inspection of physics renders, and a 180-question human baseline. These are meaningful safeguards. However, it does not report independent agreement, initial disagreements, or adjudication procedures by source and category. Camera-ray visibility checks establish visibility of queried body parts but do not by themselves establish that contact, penetration, or support judgments are unambiguous from the image under the intended assumptions. More explicit evidence linking the physical labels to observer judgments would strengthen the benchmark.
-
The estimated-state construction and evaluation protocols remain insufficiently specified for reproduction and attribution. Although Appendices C and D document operations and representation examples, the materials do not fully specify the estimation models and versions, attribute and relation generation, coordinate and scale alignment, failure handling, truncation, serialization, or question conditioning. The Image baselines also differ between Tables 2 and 5 despite using the same named model and full benchmark. Moreover, the reproducibility statement attributes prompts, decoding settings, and invalid-output handling to Section 4.1, but that section does not provide those configurations.
Questions for authors
-
Could you provide reproducible configurations for estimated-graph construction and the Image-only runs underlying Tables 2 and 5? Please specify human and object estimators, attribute and relation sources, coordinate and scale alignment, question/answer-choice conditioning, failure and truncation handling, serialization, prompts, model versions, decoding, and invalid-output handling. Please also explain the differences between the two Image baselines.
-
Building on the existing screening and human baseline, could you report independent disagreements and adjudication by source and category, and clarify whether the human-baseline annotators were independent of benchmark construction and hidden scene labels? For collision, support, and reach questions, please explain how the assumptions documented in Appendix B are communicated to respondents. A small blind audit or existing annotation records would help assess image answerability.
-
Could you separate operation selection, required-field availability, argument binding, execution outcomes, and final-answer correctness? Appendix C reports no selected operation for 717 of the 994 physical questions, with 272 of the remaining 277 calls returning UNKNOWN. Please clarify the outcomes of the five non-UNKNOWN calls and which fields oracle state supplies beyond estimated state. Without controls for field coverage, the computation conclusions should remain specific to the supplied estimated and oracle configurations.
-
Building on the grouped No-image results in Table 2 and matched feasibility probes in Table 6, could you report answer-label and answer-option distributions by category and question type, with further matched analyses where corresponding items exist? Please also qualify the abstract’s counterfactual claim: all ten open models score lower on counterfactual questions, whereas both proprietary models score slightly higher. Since descriptive and counterfactual subsets contain different questions, their score gap does not isolate the effect of counterfactual reasoning.
5 comments
Q2 & W1: Annotation screening and human evaluation
We respectfully illustrate our annotation pipeline in this section. Three annotators reviewed each candidate question and answer against its image, removing items judged ambiguous or incorrect. Screening removed 220 of 5,511 questions (3.99%), with the following category- and source-level breakdown:
| Category / Source | Entered review | Initially unanimous | Initial disagreements | Retained after adjudication | Removed | Final retained |
|---|---|---|---|---|---|---|
| Pose | 1,785 | 1,561 | 224 | 187 | 37 | 1,748 |
| Orientation | 1,256 | 1,067 | 189 | 137 | 52 | 1,204 |
| Relation | 1,376 | 1,208 | 168 | 137 | 31 | 1,345 |
| Collision | 496 | 391 | 105 | 62 | 43 | 453 |
| Kinematics | 119 | 79 | 40 | 21 | 19 | 100 |
| Intuitive physics | 479 | 387 | 92 | 54 | 38 | 441 |
| Real photographs | 3,033 | 2,639 | 394 | 276 | 118 | 2,915 |
| Rendered scenes, including photo-derived | 2,478 | 2,054 | 424 | 322 | 102 | 2,376 |
| Overall | 5,511 | 4,693 | 818 | 598 | 220 | 5,291 |
Category and source rows summarize the same question pool. These counts measure filtering, not independent inter-rater agreement or label accuracy. Four participants answered 180 stratified questions (30 per category) with two responses per question, achieving 91.4% category-reweighted overall accuracy and 90.0% on Spatio-Physics, providing a human reference on the sampled questions.
Q3: Code execution analysis
Thank you for encouraging this more detailed error analysis.
Graph and Code receive the same initial state within each estimated or oracle configuration. We therefore compare Code against its corresponding Graph baseline. Estimated-to-oracle comparisons additionally change state accuracy and field coverage.
Appendix C reports estimated Image+Code on all 994 physics questions. Below, we compare estimated and oracle Image+Code on the same 491 physics questions with available oracle-state evaluation results.
| Stage | Recorded checks | Outcome | Estimated / Oracle |
|---|---|---|---|
| Operation selection | selected_operation != "none" | Selected | 265 / 420 |
| Required-field availability | missing_required_physical_state; missing_joint_or_reach_or_contact_evidence | UNAVAILABLE; UNKNOWN | 21 / 0; 37 / 0 |
| Argument binding | cannot_parse(arguments); person_removed or all_targets_removed; friction_edit_contact_not_present | UNKNOWN for each | 17 / 7; 10 / 95; 0 / 7 |
| Execution outcomes | returned_status == "ok" | OK | 5 / 294 |
| Final-answer correctness | predicted_answer == ground_truth | Correct answer | 286/491 (58.2%) / 457/491 (93.1%) |
The five estimated-state OK returns arise because the model requests straightening the queried knees, setting their angles to 0° without needing their original values.
Oracle calls can still fail when the model supplies incorrect arguments—for example, removing an object before checking its collision—or requests a motion calculation beyond the solver’s capabilities.
Oracle state supplies additional physical evidence, including numerical contact positions and normals, gravity in the scene frame, and measured collision geometry.
The results support our configuration-specific finding that execution benefits depend on usable state and correct tool use. Holding the initial graph fixed, estimated Graph → Code gives 53.4% → 53.0%, while oracle Graph → Code gives 82.5% → 90.4%.
The estimated-graph pipeline and its input conventions are summarized below.
UNKNOWN(Appendix C, pp.15–16). Visual contact/support candidates are not treated as measured physical contacts.The Image baselines in Tables 2 and 5 use different prompts. Table 5 applies shared instructions requesting a concise, checkable reasoning explanation and the same final-answer token budget across Image, Graph and Code. We therefore rerun Image under the representation-study protocol, keeping the model checkpoint and benchmark questions unchanged. The two Image scores are consequently protocol-specific baselines, and representation gains are calculated against the corresponding Image baseline within Table 5.
We will provide experiment details in the next version of our manuscript.