acceptodds
Under review as a conference paper at ICLR 2027

SPARKBench: Best Lessons from Benchmarking Human-Centric Spatio-Physics Reasoning

Abstract

Understanding how another person can move and interact with the physical world is essential to human coordination and collaboration. Yet even a simple question—can someone bend down to pick up an object without hitting a nearby table or losing balance?—requires reasoning about articulated motion, collision, and physical stability that remains challenging for multimodal large language models. To study Spatio-Physics in humAn-centric Reasoning, we introduce SPARKBench, a benchmark with 5,291 questions over 1,442 images. It covers two macro-level categories: 3D Spatial reasoning about pose, orientation, and relations, and Spatio-Physics reasoning about collision, articulation and kinematics, and intuitive physics. We benchmark 12 vision-language models, finding that all of these models struggle to judge body orientation and movement limits while scoring lower on counterfactual than on descriptive questions. To understand how to improve human-centric spatio-physics reasoning, we compare three representations (Image, Graph, and Code) and two states, estimated from the image or taken from the rendered scene. We learn two best lessons for human-centric spatio-physics reasoning: (1) Structured body information can improve spatial judgments, but physical reasoning also needs reliable physical state relevant to the question. (2) Computing the judgment from that state can improve physical reasoning when they use reliable scene geometry and physical properties. SPARKBench with the best lessons provides a basis for identifying model weaknesses and designing better representations and computations for human-centric spatio-physics reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
benchmark, human-centric reasoning, 3D spatial reasoning, physical reasoning, multimodal large language models, structured representations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2DTuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SPARKBench, a human-centric spatio-physics reasoning benchmark containing 5,291 questions over 1,442 images. It covers 3D pose, orientation, relations, collision, articulation/kinematics, and intuitive physics, with both descriptive and counterfactual questions. Real photographs provide diverse scenes, while procedural and photo-derived renders provide stored geometry and physical state for answer verification. Questions undergo source-specific automatic checks followed by human review.

The paper evaluates twelve vision-language models and identifies weaknesses in orientation and kinematics. All ten open models perform worse on counterfactual than descriptive questions. Using Qwen3-VL-8B, the paper further compares Image, Graph, and Code representations under estimated and oracle states. Explicit body structure and derived relations improve some spatial judgments, while physical reasoning benefits from relevant state variables and executable checks on known synthetic state.

Critical strengths
  1. The benchmark connects articulated body structure with physical constraints and provides verifiable labels for part of its data. It combines body-centered reference frames, spatial relations, contact and interpenetration, joint limits, reach, and support. Procedural scenes retain geometry and physical state, with documented pose screening, deduplication, visibility checks, decision rules, and margins. Photo-derived scenes support controlled edits while explicitly treating fitted bodies as proxies rather than measured ground truth for the source photographs.

  2. The representation and execution analyses include useful negative results and diagnostic references. Estimated graphs improve some pose and orientation judgments but degrade several physical and real-image results. Operation-selection and missing-field statistics expose practical limitations of the estimated-state pipeline. Oracle comparisons demonstrate potential benefits of physical state and computation, while the authors acknowledge that state accuracy and field coverage change together. The cases also illustrate missed detections, reference-frame errors, incorrect entity binding, and failures to use available state correctly.

Critical weaknesses
  1. Image answerability and annotation reliability are only partially characterized. The paper provides geometric verification, three-annotator screening, visual inspection of physics renders, and a 180-question human baseline. These are meaningful safeguards. However, it does not report independent agreement, initial disagreements, or adjudication procedures by source and category. Camera-ray visibility checks establish visibility of queried body parts but do not by themselves establish that contact, penetration, or support judgments are unambiguous from the image under the intended assumptions. More explicit evidence linking the physical labels to observer judgments would strengthen the benchmark.

  2. The estimated-state construction and evaluation protocols remain insufficiently specified for reproduction and attribution. Although Appendices C and D document operations and representation examples, the materials do not fully specify the estimation models and versions, attribute and relation generation, coordinate and scale alignment, failure handling, truncation, serialization, or question conditioning. The Image baselines also differ between Tables 2 and 5 despite using the same named model and full benchmark. Moreover, the reproducibility statement attributes prompts, decoding settings, and invalid-output handling to Section 4.1, but that section does not provide those configurations.

Questions for authors
  1. Could you provide reproducible configurations for estimated-graph construction and the Image-only runs underlying Tables 2 and 5? Please specify human and object estimators, attribute and relation sources, coordinate and scale alignment, question/answer-choice conditioning, failure and truncation handling, serialization, prompts, model versions, decoding, and invalid-output handling. Please also explain the differences between the two Image baselines.

  2. Building on the existing screening and human baseline, could you report independent disagreements and adjudication by source and category, and clarify whether the human-baseline annotators were independent of benchmark construction and hidden scene labels? For collision, support, and reach questions, please explain how the assumptions documented in Appendix B are communicated to respondents. A small blind audit or existing annotation records would help assess image answerability.

  3. Could you separate operation selection, required-field availability, argument binding, execution outcomes, and final-answer correctness? Appendix C reports no selected operation for 717 of the 994 physical questions, with 272 of the remaining 277 calls returning UNKNOWN. Please clarify the outcomes of the five non-UNKNOWN calls and which fields oracle state supplies beyond estimated state. Without controls for field coverage, the computation conclusions should remain specific to the supplied estimated and oracle configurations.

  4. Building on the grouped No-image results in Table 2 and matched feasibility probes in Table 6, could you report answer-label and answer-option distributions by category and question type, with further matched analyses where corresponding items exist? Please also qualify the abstract’s counterfactual claim: all ten open models score lower on counterfactual questions, whereas both proprietary models score slightly higher. Since descriptive and counterfactual subsets contain different questions, their score gap does not isolate the effect of counterfactual reasoning.

5 comments

Q1 & W2: Estimated-graph construction and evaluation protocols

The estimated-graph pipeline and its input conventions are summarized below.

ComponentConfiguration
Person / object detectionViTDet-H detects people; RT-DETRv2-R50 detects objects.
Body geometrySAM 3D Body, DINOv3 variant, estimates body geometry.
Attributes / relationsQwen3-VL-4B-Instruct predicts appearance, posture, activity, visible-face information, hand observations, and candidate touching, holding and support relations.
Derived quantitiesJoint angles, facing directions and relative positions are computed deterministically from estimated geometry; query rules are described in Appendix C, Table 4, p.16.
Coordinates / scaleImage coordinates use [0,1000]; estimated 3D coordinates use a camera frame with x right, y down and z away, in metres (Appendix D, pp.17–18). Body–scene metric alignment is not independently calibrated.
Question conditioningGraph construction uses neither questions nor answer choices. Questions sharing an image use the same cached initial graph, and Graph and Code receive identical initial states. Questions and choices enter during answering and operation planning.
Serialization / missing informationGraphs are stored as JSON. Available estimates are retained; missing required fields cause operations to return UNKNOWN (Appendix C, pp.15–16). Visual contact/support candidates are not treated as measured physical contacts.

The Image baselines in Tables 2 and 5 use different prompts. Table 5 applies shared instructions requesting a concise, checkable reasoning explanation and the same final-answer token budget across Image, Graph and Code. We therefore rerun Image under the representation-study protocol, keeping the model checkpoint and benchmark questions unchanged. The two Image scores are consequently protocol-specific baselines, and representation gains are calculated against the corresponding Image baseline within Table 5.

We will provide experiment details in the next version of our manuscript.

Q2 & W1: Annotation screening and human evaluation

We respectfully illustrate our annotation pipeline in this section. Three annotators reviewed each candidate question and answer against its image, removing items judged ambiguous or incorrect. Screening removed 220 of 5,511 questions (3.99%), with the following category- and source-level breakdown:

Category / SourceEntered reviewInitially unanimousInitial disagreementsRetained after adjudicationRemovedFinal retained
Pose1,7851,561224187371,748
Orientation1,2561,067189137521,204
Relation1,3761,208168137311,345
Collision4963911056243453
Kinematics11979402119100
Intuitive physics479387925438441
Real photographs3,0332,6393942761182,915
Rendered scenes, including photo-derived2,4782,0544243221022,376
Overall5,5114,6938185982205,291

Category and source rows summarize the same question pool. These counts measure filtering, not independent inter-rater agreement or label accuracy. Four participants answered 180 stratified questions (30 per category) with two responses per question, achieving 91.4% category-reweighted overall accuracy and 90.0% on Spatio-Physics, providing a human reference on the sampled questions.

Q3: Code execution analysis

Thank you for encouraging this more detailed error analysis.

Graph and Code receive the same initial state within each estimated or oracle configuration. We therefore compare Code against its corresponding Graph baseline. Estimated-to-oracle comparisons additionally change state accuracy and field coverage.

Appendix C reports estimated Image+Code on all 994 physics questions. Below, we compare estimated and oracle Image+Code on the same 491 physics questions with available oracle-state evaluation results.

StageRecorded checksOutcomeEstimated / Oracle
Operation selectionselected_operation != "none"Selected265 / 420
Required-field availabilitymissing_required_physical_state; missing_joint_or_reach_or_contact_evidenceUNAVAILABLE; UNKNOWN21 / 0; 37 / 0
Argument bindingcannot_parse(arguments); person_removed or all_targets_removed; friction_edit_contact_not_presentUNKNOWN for each17 / 7; 10 / 95; 0 / 7
Execution outcomesreturned_status == "ok"OK5 / 294
Final-answer correctnesspredicted_answer == ground_truthCorrect answer286/491 (58.2%) / 457/491 (93.1%)

The five estimated-state OK returns arise because the model requests straightening the queried knees, setting their angles to 0° without needing their original values.

Oracle calls can still fail when the model supplies incorrect arguments—for example, removing an object before checking its collision—or requests a motion calculation beyond the solver’s capabilities.

Oracle state supplies additional physical evidence, including numerical contact positions and normals, gravity in the scene frame, and measured collision geometry.

The results support our configuration-specific finding that execution benefits depend on usable state and correct tool use. Holding the initial graph fixed, estimated Graph → Code gives 53.4% → 53.0%, while oracle Graph → Code gives 82.5% → 90.4%.

Reviewer_27ueOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SPARKBench, a benchmark of 5,291 questions on human-centric spatial and physical reasoning. Models stay far below humans on body orientation and movement. It also compares image, scene graph, and code inputs. It concludes that models should compute answers from explicit physical state.

Critical strengths

(1) The topic is useful and rarely covered. The questions link body pose and body frames to contact, joint limits, and support. Every question is reviewed by three annotators.

(2) Some findings go beyond a leaderboard. Explicit body state raises orientation accuracy from 55.6% to 72.3%. In paired feasibility probes, the model accepts only 20.6% of feasible scenes.

Critical weaknesses

(1) The two lessons are largely true by construction. The derived scene-graph fields and the code checks reuse the rules that generate the answer labels. With raw body geometry only, accuracy drops to 53.2%, below the 71.6% of the image input.

(2) Image source and text priors confound the benchmark results. Ninety of the 100 kinematics questions are synthetic SMPL renders, and Qwen3-VL-8B scores 56.8% on synthetic versus 80.6% on real 3D Spatial questions. Without the image, it still reaches 61.5% on Relation and 60.1% on Intuitive Physics, against chance levels of 39.8% and 37.3%.

Questions for authors

(1) With estimated state, the code checks return an answer for only 5 of 994 physics questions. How much of the oracle gain remains when contacts and support come from an image-based estimator?

(2) Can you report the main results separately for real and synthetic images, and on questions that a text-only model gets wrong? Do the representation results hold beyond Qwen3-VL-8B?

5 comments

W1: Are the two lessons true by construction?

We acknowledge that derived graph fields and executable checks perform part of the processing externally, using principles consistent with label verification. We respectfully clarify that the oracle graph explicitly represents scene geometry and perceptual attributes, including derived relations, as summarized below.

DomainInformation providedExample questions
SpatialPerson identifiers, joint positions and angles, head/body directions, relative positions, and depth orderWho is the cake-holder facing? After one person turns 180°, which side is the other person on?
PhysicsBody/object geometry, intersection volumes, joint constraints, centre of mass, contacts, and support conditionsCan shoulder rotation bring the hand to the target? Can balance be maintained after removing a support?

Because the benchmark covers both perception and reasoning, some descriptive questions can be answered by reading a derived field; these cases assess information retrieval and entity binding rather than multi-step reasoning. Other questions require combining information, updating relations after a specified intervention, or evaluating physical constraints. Code provides predefined checks to examine whether models can select appropriate operations, bind their arguments, and use the returned results. These checks share geometric and physical principles with label verification, so we interpret the oracle results as a controlled diagnostic of models’ use of supplied state and operations, rather than an independent validation of the checks themselves.

Additionally, the geometry-only result (53.2% versus 71.6% for Image) also changes information coverage and removes derived relations, so it does not isolate representation format. To examine the role of state quality on real photographs, we construct human-verified graphs for 300 questions from 50 real images, with annotations reviewed and refined by three annotators using a 3D visualization interface. Further annotation details are provided in our response to Reviewer qzQg’s Weakness 1.

The performance across Qwen3-VL-8B and Qwen3.5-9B are exhibited below.

ModelImageEstimated GraphEstimated CodeHuman-verified GraphHuman-verified Code
Qwen3-VL-8B62.060.761.083.089.0
Qwen3.5-9B76.375.075.382.787.3

Both models show the consistent trend: Code > Graph > Image. Relative to image configuration, human-verified Graph improves accuracy by 21.0% for Qwen3-VL-8B and 6.4% for Qwen3.5-9B. Explicit execution provides further gains of 6.0 and 4.6 points, respectively.

These results demonstrate that the benefits of structured representations and executable operations are not confined to synthetic renders. On this real-photo subset, both models benefit from human-verified scene information. This supports our interpretation that the quality and completeness of the supplied state are important limitations of the current estimated-graph pipeline.

W2: Image source and text priors confound the benchmark results

Thank you for your constructive suggestions. We are grateful to have a chance to clarify these concerns.

  1. Contribution from the text prior

To reduce the contribution from the text prior, we evaluate a fixed subset of 2,588 QA answered incorrectly by the no-image baseline (Qwen3-VL-8B-Instruct):

Task / StateNComparisonAccuracy (%)
Pose / Estimated934Image → Graph59.4 → 74.9
Orientation / Estimated654Image → Graph47.2 → 66.2
Relation / Estimated518Image → Graph85.7 → 66.2
Spatio-Physics / Estimated482Image → Graph64.7 → 47.7
3D Spatial / Oracle1,011Graph → Code88.3 → 89.1
Spatio-Physics / Oracle269Graph → Code79.6 → 85.1
Intuitive Physics / Oracle32Graph → Code71.9 → 96.9

As shown in the table above, estimated Graph improves over Image by 15.5% and 19.0% on Pose and Orientation, while oracle Code improves over Graph by 5.5 points on Physics and 25.0 points on its 32-question Intuitive Physics subset. These results show that representation and execution gains persist on questions that the no-image baseline fails to answer, supporting both lessons beyond this baseline’s text-only performance. We will clarify the role of text priors and include these additional analyses in the revised manuscript.

  1. Contribution from the image prior

We agree that estimated-graph benefits differ between real and synthetic images. The lower Image accuracy on synthetic renders partly reflect VLM sensitivity to unfamiliar visual appearances. This motivates representations that support more consistent human-body reasoning across image domains.

On real images, estimated Graph and Code improve orientation accuracy from 68.2% to 69.6% and 71.6%, respectively, but reduce physics accuracy from 87.9% to 73.3% and 73.1%. We also highlights the importance of accurate, task-relevant state in realizing the benefits of structured representations in paper.

As detailed in our response to W1, providing refined scene graphs improves performance on the same questions from a fixed subset of 50 real photographs. The gains on real photographs support our lesson that accurate, task-relevant state can improve human-body reasoning beyond synthetic renders.

We thank the reviewer for highlighting these two issues and will include the complete source- and category-specific results in the revised manuscript.

Q2: Results by image source, performance on text-only errors, and generalization across models

Separate results for real and synthetic images. We respectfully report the main results separately for real photographs, procedural synthetic images, and photo-derived renders, keeping the latter as a distinct source.

Full-benchmark accuracy (%):

ModelAll (N=5,291)Real (N=2,915)Synthetic (N=2,246)Photo-derived (N=130)
GPT-582.887.676.879.2
Claude Opus 4.882.787.176.985.4
Qwen3-VL-4B70.781.157.665.4
Qwen3-VL-8B71.481.358.866.2
Qwen3-VL-30B-A3B72.384.357.363.1
Qwen3-VL-8B-Thinking72.283.259.054.6
InternVL3.5-8B66.473.858.340.8
Gemma3-27B67.875.857.760.0
HumanVLM 8B62.375.944.856.9
SpatialRGPT 8B (RGB)66.875.555.468.5
SpatialReasoner53.060.043.954.6
Qwen3.5-4B69.273.064.861.5

All models score higher on real photographs than on procedural synthetic images, although these differences do not isolate source effects because the subsets also differ in question composition.

We also evaluate all models on one fixed set of 2,588 questions answered incorrectly by Qwen3-VL-8B-Instruct without images, using the same selected questions across models.

Accuracy (%) on the fixed text-only-error subset

ModelAll (N=2,588)Real (N=1,278)Synthetic (N=1,244)Photo-derived (N=66)
GPT-578.082.573.575.8
Claude Opus 4.877.081.971.584.8
Qwen3-VL-4B59.373.644.954.5
Qwen3-VL-8B58.670.846.453.0
Qwen3-VL-30B-A3B63.877.949.854.5
Qwen3-VL-8B-Thinking61.275.846.751.5
InternVL3.5-8B53.166.040.837.9
Gemma3-27B54.563.745.448.5
HumanVLM 8B50.467.831.963.6
SpatialRGPT 8B (RGB)57.166.746.668.2
SpatialReasoner47.052.741.051.5
Qwen3.5-4B61.262.660.350.0

GPT-5 and Claude Opus 4.8 achieve 78.0% and 77.0% on this subset, showing that their correct answers are not confined to questions already solved by the selected text-only baseline; this filtering does not eliminate all text-prior effects.

Beyond Qwen3-VL-8B, we extend the representation study to Claude Opus 4.8, Gemma3-27B, and Qwen3.5-9B on supplementary subsets of 300 synthetic and 300 real-photo questions.

Representation accuracy (%) on 300 synthetic questions

ModelImageEstimated GraphEstimated CodeOracle GraphOracle Code
Claude Opus 4.879.080.378.084.789.0
Gemma3-27B54.769.065.074.079.3
Qwen3.5-9B60.075.074.382.388.0

Representation accuracy (%) on 300 real-photo questions

ModelImageEstimated GraphImage + GraphEstimated CodeImage + Code
Claude Opus 4.887.081.090.384.389.7
Gemma3-27B74.074.778.370.079.3
Qwen3.5-9B82.074.382.373.084.3

On synthetic questions, oracle state improves Graph accuracy across all three models by 4.4–7.3 points, and Code adds 4.3–5.7 points with oracle state, whereas it reduces accuracy by 0.7–4.0 points with estimated state.

On real images, estimated representations alone do not consistently outperform images, but Image+Code improves over Image across all three models by 2.3–5.3 points.

These results extend the evidence beyond one model while supporting conditional conclusions: the supplied state matters, computation benefits depend on that state, and structured information can complement real-image evidence.

Reviewer_qzQgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SPARKBench, a benchmark of 5,291 multiple-choice questions over 1,442 images about how people are arranged in space and what they can physically do. 3D Spatial covers pose, orientation and relations (4,297 questions). Spatio-Physics covers collision, kinematics and intuitive physics (994 questions). About 6% of the questions are counterfactual and ask about the outcome of a stated change.

The images come from three sources.

  • 593 real photos from COCO and the web, with human-written questions filtered by a consistency check under horizontal flips and blur.
  • 800 procedural SMPL renders with template questions, whose answers are checked against the stored geometry with explicit margins for facing, joint angles, depth, static equilibrium, interpenetration and reach.
  • 49 photo-derived scenes in which a fitted SMPL body is re-rendered with one variable changed.

Three annotators then review all questions and keep 96% of them.

Twelve models are evaluated. GPT-5 and Claude Opus 4.8 reach about 83% overall against 91% for humans, and the ten open models reach 53 to 72%. All open models score lower on orientation than on relations, lower on kinematics than on intuitive physics, and lower on counterfactual than on descriptive questions.

The paper then studies Qwen3-VL-8B with three input representations. These are the image, a scene graph (Graph), and the same graph with executable checks such as static equilibrium, collision and reach (Code). The graph holds either a state estimated from the image or the oracle state from the renderer. From this study the paper draws two lessons. Structured body information helps spatial judgments, but physical reasoning needs a reliable physical state. Computing the judgment from that state then helps when the state is reliable.

Critical strengths
  • The synthetic labels are rigorous by eachanswer being checked against the stored meshes, joints, contacts and camera. It must also clear an explicit margin from the decision boundary, such as 15 degrees for facing or 20 mm for the centre of mass, and the queried body part must be visible.
  • The benchmark targets a real gap. Existing human-centric benchmarks cover pose, attributes and relations, and physics benchmarks cover objects. SPARKBench evaluates whether a person can reach a target, would fall, or interpenetrates another body, with person-centric left and right and counterfactual edits. The evaluation exposes a consistent weakness on orientation and kinematics. The best proprietary models score 62 to 69% on these two categories, against 87 and 92% for humans.
  • Comparing estimated and oracle state is a useful diagnostic. Keeping the model and question fixed while changing where the scene state comes from separates perception errors from reasoning errors. The case studies make the failure modes concrete. In one, a missed bed leads the model to conclude that a person is sitting on the floor.
Critical weaknesses
  • On real photos, the results contradict the first finding. The paper reports that on real images the image alone reaches 80.6% on 3D Spatial, against at most 71.3% with the estimated Graph or Code. Subtracting the synthetic results from the full-set results, the estimated graph lowers pose accuracy from about 75% to 69% and relation accuracy from about 92% to 72% on non-synthetic questions. It raises orientation by only roughly2 points. The spatial gains come from the synthetic renders, which are untextured bodies on a blank background. On those renders the image-only baseline is weak, at 55% on pose and 45% on orientation.
  • Geometry-checked answers exist only for the renders and the 49 photo-derived scenes. Physics questions on real photos are labeled by human judgment. The flip and blur consistency check only tests left/right logic, not whether a physics answer is correct.
  • All lessons come from one model. The representation study uses only Qwen3-VL-8B, with no variance estimates. Even the image-only accuracy of this model differs by up to 4.4 points between the main evaluation and the representation study (73.1% against 68.7% on collision), which is as large as several of the reported effects. A single 8B model does not generally suffice as best lessons framing.
  • The paper does not say which body estimator, detector or model produces the estimated joints, contacts and text fields such as appearance, activity and visible face. Without this information the representation results cannot be interpreted or reproduced.
  • The physics and counterfactual subsets are small. Kinematics has 100 questions, synthetic intuitive physics 67, and counterfactuals 324, and the human baseline uses 30 questions per category. No confidence intervals are reported, so differences of a few points cannot be judged.
Questions for authors

Q1: The paper does not say how the estimated scene graph is built. Which models produce the body geometry, the object detections and the text fields such as appearance and activity? Q2: The oracle graph and the Code operations seem to use the same quantities and rules that produced the labels. Which oracle fields are given for each category, and how often do they contain the answer directly? Q3: Do the lessons hold beyond Qwen3-VL-8B? Repeating the representation study with at least one other open model and one proprietary model would show whether they generalize.

7 comments

W1: Real-image results with human-verified graphs

Thank you for pointing this out.

We respectively argue that the performance drop primarily attributes to limitations of the current perception pipeline, instead of the difference between the synthetic data and realistic data. The estimated graphs introduce visual errors, which is harmful to the model reasoning.

To better illustrate this, we annotate 50 images from the 593 real images and construct graphs through a human-in-the-loop annotation pipeline. We firstly define the scene attributes required for annotation, and use GPT6-Astra to generate initial annotations. Three annotators then review and refine these annotations using a dedicated 3D visualization software. Images containing annotation errors are excluded, and the reviewed annotations are consolidated into final scene graphs.

These 50 images yield 300 QA pairs. With these annotations, we evaluate Qwen3-VL-8B and Qwen3.5-9B under three input configurations:

  • image, which provides the original photo only;
  • graph, which provides the human-verified scene graph;
  • code, which provides the same graph together with executable operations.

All these configurations are evaluated on the same questions.

ModelImageEstimated GraphEstimated CodeHuman-verified GraphHuman-verified Code
Qwen3-VL-8B62.060.761.083.089.0
Qwen3.5-9B76.375.075.382.787.3

Both models show the consistent trend: Code > Graph > Image. Relative to image configuration, human-verified Graph improves accuracy by 21.0% for Qwen3-VL-8B and 6.4% for Qwen3.5-9B. Explicit execution provides further gains of 6.0 and 4.6 points, respectively.

These results demonstrate that the benefits of structured representations and executable operations are not confined to synthetic renders. On this real-photo subset, both models benefit from human-verified scene information. This supports our interpretation that the quality and completeness of the supplied state are important limitations of the current estimated-graph pipeline.

W2: Human review and annotation quality control

We agree that flip / blur consistency does not establish physical correctness. To reduce the annotation error introduced by humans, each candidate question and its answer are reviewed against the corresponding image by three annotators. Questions judged ambiguous or incorrect are removed. As summarized below, human screening removes 220 of 5,511 questions (3.99%), retaining 5,291 (96.01%). The removal rates are 2.72% for 3D Spatial and 9.14% for Spatio-Physics. These statistics document the extent of human filtering, rather than inter-rater agreement or label accuracy. Real-photo labels therefore remain human-reviewed judgements, while rendered-scene labels receive additional geometry checks.

DomainReviewedRemovedRemoval rateRetainedRetention rate
3D Spatial4,4171202.72%4,29797.28%
Spatio-Physics1,0941009.14%99490.86%
Overall5,5112203.99%5,29196.01%

W3 & Q3: Additional evaluations across more models

To address your concerns, we extend the study to three additional models (Claude Opus 4.8, Gemma3-27B, and Qwen3.5-9B) on a 300 question synthetic dataset and real dataset. On synthetic questions, oracle state improves graph accuracy across all three models by 4.4-7.3%. Code provides further gains of 4.3-5.7 points with oracle state, but little gains with estimated state. Real subset (using estimated state and separate from the human-verified images subset in W1) exhibit the consistent trend. Image + code outperforms image across all three models by 2.3-5.3 points. These results extend the evidence beyond one model, highlight that the reliable state helps and code gains depend on the supplied state, and structured information can complement real-image evidence.

Performance on 300 Synthetic QA.

ModelImageEstimated GraphEstimated CodeOracle GraphOracle Code
Claude Opus 4.879.080.378.084.789.0
Gemma3-27B54.769.065.074.079.3
Qwen3.5-9B60.075.074.382.388.0

Performance on 300 Real QA.

modelImageEstimated Graphgraph+ImageEstimated CodeImage+Code
Claude Opus4.887.081.090.384.389.7
Gemma3-27B74.074.778.370.079.3
Qwen3.5-9B82.074.382.373.084.3

Note that, the details of the real dataset oracle graph collection are provided in the response to Weakness 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.