acceptodds
Under review as a conference paper at ICLR 2027

Rulers, Not Meters: Discovering Falsifiable World Models through Active Visual Experimentation

Abstract

A useful world model must predict the effects of new interventions, so it has to be identified, not merely fit. Existing evaluations of vision-language agents fix what the agent observes and cannot tell identification from fitting guesses to feedback. We study active world-model identification in Eyes of Galileo: from raw renders of 35 counterfactual physics scenes, an agent must build its own measuring references, propose an executable symbolic world model, and revise it when execution at dispersed targets exposes counterexamples. With GPT-5.6, such an agent identifies 63.4% of hidden world models (strict or relaxed), versus at most 44.6% for three visual-reasoning scaffolds under identical feedback, and dispersed counterexamples raise exact identification from 33% to 73% in a controlled revision study. The ability depends on the backbone (33.1% with Qwen3-VL-32B, 24.0% with Qwen3-VL-8B), and so does learning it: fine-tuning on successful GPT-5.6 episodes raises the 32B model's success on unseen scenes from 42.9% to 57.1%, but does not help the 8B model. Identifying world models from pixels thus hinges on both making evidence and learning to use it, a capability that fixed-observation evaluations cannot reveal.

Primary area:
foundation or frontier models, including LLMs
Keywords:
world models, scientific discovery, vision-language agents, active experimentation, physical law discovery, falsification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_A7saOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether VLM agents can discover unknown physical laws through active visual experimentation. It introduces Eyes of Galileo, a benchmark of 35 counterfactual physics scenes, and an Observation-Feedback Discovery framework in which agents configure visual measurement tools, propose symbolic laws, and revise them based on execution feedback at multiple targets. The experiments examine the effects of visual instruments, feedback strategies, model capability, and fine-tuning on law recovery. The results show that visual instruments reduce input-token costs, while the benefits of multi-target feedback and fine-tuning vary across model backbones.

Critical strengths

The paper introduces a potentially useful evaluation setting for visual scientific discovery. Agents must configure visual measurement tools, propose symbolic laws, and test them through interventions, making the acquisition of measurement evidence part of the task.

Critical weaknesses
  1. Unclear contribution and overly broad framing. The paper combines a benchmark, an agent workflow, and fine-tuning experiments, but does not clearly establish the methodological contribution of OFD over existing interactive law-discovery approaches. The task scope is also limited: each scene focuses on one control–observable relation, and 30 of the 35 scenes use power laws. This provides limited support for broader claims about world-model identification.

  2. The relaxed recovery criterion does not ensure predictive accuracy. As written, a predicted exponent passes if it is closer to the counterfactual value than to the textbook value. For example, a prediction of 10 would satisfy this condition when the true exponent is 3 and the textbook exponent is 2, despite being highly inaccurate. The headline 63.4% recovery rate therefore does not necessarily indicate successful law identification; Strict recovery is only 23.4%. The criterion needs clearer justification and implementation details.

  3. The central contribution of visual instruments is insufficiently supported. The instrumented agent achieves 23.4% Strict and 63.4% S/R recovery, compared with 25.7% and 61.1% without instruments; neither difference is significant. These results do not establish that constructing measurement references improves law recovery. The token savings are useful, but their attribution to measurement quality is unclear because image sizes also differ between conditions.

  4. The role of active experimentation remains unclear. In the instrumented condition, 95.7% of consecutive law submissions involve no additional active observation. Agents also receive an oracle law-fit signal derived from the evaluation grid. The experiments do not sufficiently separate the contribution of active measurement selection from repeated formula revision using this feedback.

  5. The feedback experiment does not isolate the claimed mechanism. The improvement from 33.3% to 73.3% comes from a single-revision experiment using numeric feedback. Comparing one target with three simultaneously changes target coverage and feedback quantity. This result cannot be attributed specifically to dispersed counterexamples or directly generalized to the full visual interaction setting.

  6. The method’s advantage is inconsistent across models and metrics. With GPT-5.6, the S/R improvement does not translate into superior Strict recovery: 23.4% versus 23.8% for VideoSearcher. With Qwen3-VL-8B, the proposed framework underperforms all three baselines on both recovery metrics. No corresponding baseline comparisons are reported for Qwen3-VL-32B. These results limit the evidence for a general advantage of the proposed workflow.

Questions for authors
  1. Clarification of the scoring rule. Does the implemented relaxed-recovery criterion accept the exponent example described above? Are there additional constraints on absolute exponent error, coefficient accuracy, or predictive error? How many reported S/R successes rely solely on the “closer to the counterfactual than to the textbook value” condition?

  2. Evidence for the measurement contribution. Given the similar recovery rates with and without instruments, what evidence isolates the contribution of active visual measurement? In particular, are there ablations that remove the oracle law-fit signal while retaining physical-execution feedback?

  3. Evidence for the feedback mechanism. Does the benefit of dispersed targets persist with visual feedback and a matched number of targets? A comparison between equally sized sets of concentrated and dispersed targets would help determine whether coverage itself explains the reported improvement.

9 comments

Response to Critical Weakness 1

Critical Weakness 1. Unclear contribution and overly broad framing. The paper combines a benchmark, an agent workflow, and fine-tuning experiments, but does not clearly establish the methodological contribution of OFD over existing interactive law-discovery approaches. The task scope is also limited: each scene focuses on one control–observable relation, and 30 of the 35 scenes use power laws. This provides limited support for broader claims about world-model identification.

Response. We agree that the submission combines several components without separating their roles clearly enough. We will revise the framing around two scoped contributions: (i) Eyes of Galileo, an executable benchmark in which an agent must acquire visual evidence, submit a symbolic law, and test that law through physical interventions; and (ii) OFD, an interaction protocol that exposes task-aligned visual measurement operations and couples proposed laws to target-conditioned physical execution and revision. We do not claim a new general-purpose symbolic-regression algorithm or general world-model identification.

We will also narrow the task-scope claims. The original benchmark varies one exposed control at a time. Our audit identifies 25 simple-power scenes and 10 non-simple families, including exponential decay and cooling, stretched-exponential damping, affine and logarithmic responses, generalized lens, trigonometric refraction and polarization, and resonance. Thus, the benchmark is not limited to 30 power-law scenes, but the reviewer's broader point remains valid: the original benchmark alone does not establish arbitrary multivariate world-model identification.

To test that limitation directly, we added a separate paired 80-law multi-control stress test covering multivariate free fall, projectile motion, exponential cooling, and rational collision dynamics. Every law contains multiple active controls plus a distractor, and complete expressions are evaluated on 24 unseen multidimensional control combinations.

OutcomeNo instrumentsInstrumentsPaired exact pp
Correct variable set63/80 (78.8%)68/80 (85.0%).332
Structural recovery10/80 (12.5%)32/80 (40.0%)1.13×10−41.13\times10^{-4}
Held-out predictive fit10/80 (12.5%)39/80 (48.8%)8.96×10−68.96\times10^{-6}

We will report this as a separate stress test, not as proof of general multivariate discovery. The revised claim is that the benchmark and protocol test visually grounded, intervention-based law discovery, with evidence across both single-control and selected multi-control families.

Response to Critical Weakness 2

Critical Weakness 2. The relaxed recovery criterion does not ensure predictive accuracy. As written, a predicted exponent passes if it is closer to the counterfactual value than to the textbook value. For example, a prediction of 10 would satisfy this condition when the true exponent is 3 and the textbook exponent is 2, despite being highly inaccurate. The headline 63.4% recovery rate therefore does not necessarily indicate successful law identification; Strict recovery is only 23.4%. The criterion needs clearer justification and implementation details.

Response. The reviewer is correct about the specific failure case: the implemented relative-to-textbook clause has no absolute-error cap and would accept the stated prediction. We therefore rescored all 175 OFD outputs and replaced the single S/R label with separate measures of structural recovery, absolute parameter recovery, and predictive fit.

Revised evaluation measureResult
Correct variable/function structure106/175 (60.6%)
Correct power structure + exponent error ≤.30\leq .3042/125 (33.6%)
Correct power structure + exponent error ≤.20\leq .2041/125 (32.8%)
Correct power structure + exponent error ≤.10\leq .1037/125 (29.6%)
Strict predictive fit (JLR ≤.20\leq .20)41/175 (23.4%)

The 125-cell denominator contains the simple-power scenes and requires both the correct functional structure and the stated absolute exponent tolerance. The 175-cell structural score ignores numerical parameters and requires the correct variables and function family. The former “Strict” score is renamed strict predictive fit, because finite-grid JLR does not establish exact symbolic recovery.

We also decomposed the submitted 63.4% S/R result. The original OR rule counts 111/175 cells. Removing the relative-to-textbook clause, while retaining strict predictive fit or absolute exponent error ≤.30\leq .30, reduces this to 85/175 (48.6%). Thus, 26/175 cells rely solely on the relative-to-textbook clause. There is no separate coefficient-accuracy constraint in the relaxed branch; coefficient and full predictive calibration are enforced only through the strict predictive-fit branch.

We will retain the submitted S/R number only for transparency and relabel “closer than textbook” as an improvement-over-prior diagnostic, not law recovery. The revised paper will use structural recovery, absolute parameter recovery, and strict predictive fit as distinct primary outcomes and will no longer describe 63.4% as exact law identification.

Response to Critical Weakness 3

Critical Weakness 3. The central contribution of visual instruments is insufficiently supported. The instrumented agent achieves 23.4% Strict and 63.4% S/R recovery, compared with 25.7% and 61.1% without instruments; neither difference is significant. These results do not establish that constructing measurement references improves law recovery. The token savings are useful, but their attribution to measurement quality is unclear because image sizes also differ between conditions.

Response. We agree only with the narrower observation that the original 35-scene aggregate does not show a uniform recovery gain from instruments. We respectfully disagree with the broader conclusion that constructed measurement references do not improve law recovery. The original benchmark is dominated by one-control scenes in which the relevant variable is already exposed and many laws can be inferred without quantitative measurement, making it a weak test of the instrument hypothesis.

We therefore added a paired 80-law stress test designed specifically to isolate when measurement references should matter. It covers multivariate free fall, projectile motion, exponential cooling, and rational collision dynamics. Every hidden law contains multiple active controls plus a distractor, and every recovered expression is evaluated on 24 unseen multidimensional control combinations. The frozen GPT-5.6 model, prompt scaffold, action budget, physical-execution feedback, submission schema, and scorer are identical between paired arms; the only difference is whether scene-relevant visual measurement instruments are available.

OutcomeNo instrumentsInstrumentsPaired exact pp
Correct variable set63/80 (78.8%)68/80 (85.0%).332
Structural recovery10/80 (12.5%)32/80 (40.0%)1.13×10−41.13\times10^{-4}
Held-out predictive fit10/80 (12.5%)39/80 (48.8%)8.96×10−68.96\times10^{-6}

The instruments do not significantly change variable selection, but they produce a 3.2-fold increase in structural recovery and a 3.9-fold increase in held-out predictive fit. Both gains are highly significant under paired exact tests. Because the evaluation uses complete symbolic structure and unseen multidimensional combinations, these results cannot be explained by the original relaxed exponent rule or by fitting a one-dimensional evaluation grid.

The effect is task dependent rather than universal: free fall shows little benefit, whereas projectile, cooling, and collision scenes benefit substantially from quantitative references. This heterogeneity explains why the original aggregate, which contains many simple one-control scenes, dilutes the measurement contribution. It also supports a more precise methodological claim: visual instruments are valuable when competing hypotheses require quantitative geometric, temporal, or response measurements; they are not expected to help when the law is already identifiable from the exposed control and raw observation.

The original benchmark also shows an operational efficiency benefit: the instrumented condition uses 23.2% fewer input tokens per successful discovery and exhibits fewer formula-family changes. We agree that image-size differences prevent attributing the entire token reduction solely to measurement quality, so we will describe this as an end-to-end efficiency result rather than a pure causal estimate of reference quality.

Accordingly, we will revise the paper to replace the overly broad claim that instruments uniformly improve all scenes with the supported claim that scene-relevant measurement references significantly improve structural and held-out recovery in measurement-dependent multivariate tasks, while also improving end-to-end efficiency on the original benchmark.

Reviewer_UjcDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a vision-language agent can actively identify executable physical laws from rendered images alone, without access to simulator states or numeric ground-truth tables. The authors introduce Eyes of Galileo, a counterfactual physics benchmark whose hidden parameters depart from textbook defaults to rule out rote memorization, together with the Observation-Feedback Discovery (OFD) loop: the agent constructs dynamics-preserving visual references (such as grids or strobe trails), submits an executable symbolic law, and revises it using counterexamples from goal-conditioned execution (GCE). Experiments compare OFD against visual-tool agent baselines across proprietary and open-weight vision-language backbones, accompanied by ablations on visual measuring instruments, feedback formats, and trajectory-based fine-tuning.

Critical strengths
  1. The task design rules out scoring by formula memorization. Hidden parameters in the 35 scenes depart from textbook values, and the model sees only images and must submit a formula the environment can execute. Using the textbook exponent with a coefficient fitted to the true outcomes reaches only 14.3% lenient success across the 175 items, and 0 on the settings far from the textbook; on that same far group, GPT-5.6 still reaches 67.1% (lenient success means the whole formula is accurate enough or the power-law exponent is close enough). The reported gains therefore reflect recovering exponents from the scene rather than recalling textbook formulas.

  2. The experimental controls are thorough: they show a clear lenient-metric gain for the full procedure while retaining negative ablations. With matched items, step budgets, three-target pass/fail feedback, and scoring, GPT-5.6 reaches 63.4% lenient success (111/175), and the lower end of its scene-cluster interval, 53.1%, remains above the best of the three visual-tool baselines at 44.6%. Separate tables test instruments on versus off, one target versus three, 8B versus 32B, and numeric versus rendered feedback, making it possible to separate the effect of the full scaffold from the effect of each component.

Critical weaknesses
  1. The visual measuring tools (“rulers”) do not improve law-identification accuracy and therefore do not explain the main-table lead over the baselines. Table 7 shows that turning off every measuring tool so the agent sees only raw renders gives 61.1% lenient success and 25.7% strict success, compared with 63.4% and 23.4% with instruments (p=0.63p=0.63). The three baselines also receive three-target pass/fail feedback. The 18.8-point lenient lead in Table 1 (63.4% versus 44.6%) therefore comes from the surrounding prompting and revision scaffold rather than from building visual references. What the instruments actually improve is cost and reading stability: 23.2% fewer input tokens per lenient success, fewer formula-skeleton changes, and stable readings in the Section 6 cases. Yet the abstract, introduction, and conclusion still attribute the main-table accuracy gap to constructed references.

  2. Strict accuracy does not beat the baseline, and the abstract's 33% to 73% jump comes from numeric feedback rather than the main visual loop. In the main visual experiment, strict success is 23.4%, matching VideoSearcher's 23.8%, and mean error is 2.393 against 1.974. The 63.4% lenient score includes formulas whose coefficients remain inaccurate and that pass only because the power-law exponent is close enough. The abstract's jump from 33.3% to 73.3% comes from the 20-item single-revision study in Table 6, where numeric outcomes are shown directly to the model; Appendix F.4 and Appendix I state that this effect size cannot be transferred directly to the multi-turn visual loop. In Appendix E.3, on matched revision prompts, Qwen3-VL-32B reaches 16.8% strict success with numeric outcomes versus 3.9% with the main loop's image-plus-pass/fail feedback. Moreover, 95.7% of consecutive submissions in the main loop occur without a new observation, and the tool response exposes an overall law-fit flag computed on the scoring grid.

Questions for authors

Questions whose answers could change the assessment:

  1. What is the exact pass condition for the oracle law-fit flag in the tool response, and does it use the evaluation reference grid and the Strict threshold? If that flag is removed so the agent receives only execution renders and per-target pass/fail on the three targets, what are GPT-5.6's Strict and S/R scores? The pendulum trace in Appendix D shows that the agent uses this flag during revision.
  2. What are the paired S/R, Strict, and mean-JLR differences between no-instrument OFD and DeepEyesV2, Visual Sketchpad, and VideoSearcher? Do the three baselines share the exact same law-submission schema, revision prompt, and law-fit flag? Because no-instrument OFD already reaches 61.1% S/R, the Table 1 gap should be attributed explicitly to the surrounding interaction scaffold.
  3. If the numeric feedback in Table 6 is replaced by the main loop's execution renders plus per-target pass/fail, with target count or feedback volume matched, does a single revision still raise Strict recovery from about 33% to about 73%? Appendix F.4 notes that this control is required before transferring that effect size to the visual loop.

Future work, not requested as a condition for changing this rating:

  • Evaluate the three visual-tool baselines with Qwen3-VL-32B.
  • Add a preregistered second fold and multiple seeds for the 32B end-to-end fine-tuning experiment.
5 comments

Response to Weakness 1: What do the visual instruments actually contribute?

Thank you for separating the contribution of the visual instruments from that of the surrounding interaction scaffold. We agree that the original 35-scene ablation does not support the claim that constructed references alone explain the main-table accuracy lead. Removing all measuring instruments changes S/R only from 63.4% to 61.1% and is not statistically significant (p=.63p=.63); strict predictive fit changes from 23.4% with instruments to 25.7% without them. Moreover, the three visual baselines also receive the same three-target execution and per-target PASS/FAIL feedback. Therefore, the original 18.8-point gap between full OFD and the strongest visual baseline (63.4% versus 44.6%) cannot be attributed to visual instruments alone.

Separating the interaction scaffold from the visual operation

We revisited the baseline implementation to make this comparison more explicit. The original artifacts did not preserve fully auditable adapters, prompts, and traces for DeepEyesV2, Visual Sketchpad, and VideoSearcher. We therefore rebuilt all three inside the same Galileo harness. They now share the frozen GPT-5.6 backbone and reasoning configuration, scientific prompt, run_experiment, request_more_scenes, submit_law, 10-action budget, three-target execution feedback, law schema, law-fit-flag setting, termination rule, and scorer. The only method-specific difference is the visual operation:

  • DeepEyesV2 receives crop/zoom and grayscale/edge inspection;
  • Visual Sketchpad receives bounded line/box/circle annotation;
  • VideoSearcher receives temporal-segment search and key-frame extraction.

In an interface and behavior audit over wave-string, single-slit, and resonance scenes, all three adapters were correctly exposed and executable. Nevertheless, the agents elected not to invoke their method-specific visual operation and instead relied on the shared experiment–submit–counterexample–revise loop. We do not treat this smoke audit as a new formal performance comparison, but it provides direct behavioral evidence for the reviewer's interpretation: the domain-aligned interaction scaffold is already useful, whereas generic crop, sketch, and video-search operations may provide little marginal information for identifying a control–outcome function under a limited action budget.

Accordingly, we will revise the original-paper attribution. The main-table accuracy gain will be attributed to the complete interaction-and-falsification scaffold. The original benchmark supports instrument benefits in a different form: 23.2% fewer input tokens per successful discovery, fewer formula-skeleton changes, and more stable visual readings in the Section 6 cases. We will state these efficiency and stability benefits explicitly and remove language in the abstract, introduction, and conclusion that attributes the original 18.8-point accuracy gap to constructed references.

New paired evidence on measurement-intensive hidden laws

We further tested whether task-aligned instruments improve accuracy when the hidden law is multivariate and the textbook parameterization is unreliable. We ran 80 paired hidden laws over free-fall, projectile, multivariate exponential cooling, and rational collision families. Every law contains multiple active controls and a distractor, and each submitted expression is evaluated on 24 unseen multidimensional control combinations. The frozen model, prompt scaffold, 10-action budget, execution feedback, and scorer are identical across conditions; only scene-relevant measurement instruments are made available.

OutcomeNo instrumentsInstruments availablePaired exact pp
Correct variable set63/80 (78.8%)68/80 (85.0%).332
Structural recovery10/80 (12.5%)32/80 (40.0%)1.13×10−41.13\times10^{-4}
Held-out predictive fit10/80 (12.5%)39/80 (48.8%)8.96×10−68.96\times10^{-6}

The new experiment shows that instrument availability does not significantly change variable selection, but significantly improves recovery of the functional structure and complete prediction on unseen multidimensional combinations. The family breakdown is also informative: free fall does not benefit, whereas projectile, cooling, and collision do. Instruments are therefore not a universally accuracy-improving component; their value depends on whether structured measurement resolves evidence needed to distinguish competing laws.

Our revised conclusion is thus narrower and better supported: the interaction scaffold explains the original main-table lead; visual instruments improve efficiency and reading stability on the original benchmark; and task-aligned instruments can additionally improve structural and predictive recovery on measurement-intensive multivariate laws.

Response to Weakness 2: Strict recovery, the relaxed metric, and numeric-feedback revision

Response. We agree that the former Strict/Relaxed terminology conflates distinct outcomes. The former Strict criterion, JLR ≤.20\leq .20 on a finite scene-specific grid, measures tightly calibrated prediction over that range rather than exact symbolic recovery. On the original benchmark, OFD obtains 23.4%, versus 23.8% for VideoSearcher. We therefore rename this metric strict predictive fit and do not claim an original-benchmark advantage under it.

Factorized recovery metrics

We now separate structural recovery, parameter recovery, and held-out predictive fit. Rescoring the archived OFD outputs gives:

OutcomeResult
Correct variable/function structure106/175 (60.6%)
Correct power structure + exponent error ≤.30\leq .3042/125 (33.6%)
Correct power structure + exponent error ≤.20\leq .2041/125 (32.8%)
Correct power structure + exponent error ≤.10\leq .1037/125 (29.6%)

The parameter rows concern the 125 simple-power cells and require correct structure plus the stated absolute tolerance. Thus, recovering the functional structure is substantially more common than calibrating the exponent precisely.

The submitted relaxed score also contains a “closer than the textbook default” clause. Tightening only the exponent tolerance has little effect because this relative clause remains active:

Relaxed scoring ruleδ=.30\delta=.30δ=.20\delta=.20δ=.10\delta=.10
Existing S/R OR rule111/175 (63.4%)110/175 (62.9%)110/175 (62.9%)

We will therefore retain these submitted numbers for transparency but report “closer than textbook” only as an improvement-over-prior diagnostic, not as law recovery. Because baseline final expressions were not retained, we do not make a new baseline-ranking claim under structural recovery.

Beyond one-control exponent fitting

The new 80-law paired stress test does not use the relaxed OR rule. It covers free-fall, projectile, multivariate exponential cooling, and rational collision families; every law contains multiple active controls plus a distractor, and every complete expression is evaluated on 24 unseen multidimensional combinations.

OutcomeNo instrumentsInstrumentsPaired exact pp
Correct variable set63/80 (78.8%)68/80 (85.0%).332
Structural recovery10/80 (12.5%)32/80 (40.0%)1.13×10−41.13\times10^{-4}
Held-out predictive fit10/80 (12.5%)39/80 (48.8%)8.96×10−68.96\times10^{-6}

Thus, the new conclusion does not rely on the original relaxed metric: instruments significantly improve recovery of the multivariate form and complete prediction on unseen combinations. We do not claim superiority over the three visual baselines on this new benchmark because a full baseline matrix was not run.

Numeric versus faithful visual revision

We also agree that the abstract overgeneralizes the 33.3%→73.3% result. The main text already states that it is a matched numeric-feedback, one-revision study, and the appendix states that its 60 rows per arm comprise only 20 independent scene–setting cells. We will remove this number from the abstract and qualify it in the introduction.

The submission already includes a same-prompt modality control using 588 Qwen3-VL-32B revision prompts:

One-step revision outcomeFaithful rendersNumeric outcomes
Usable revisions587528
Verified: JLR ≤.20\leq .20 + correct sign3.9%16.8%
Normalization error reduced64%70%
Median normalization error1.31→0.861.28→0.57
Shape error reduced24%52%
Median shape error0.26→0.310.24→0.20

Faithful feedback contains the goal, per-target PASS/FAIL, inverted control, and three execution renders, but no outcome values. These results show that renders often support rescaling but are substantially weaker than numbers for correcting functional form. This control is not an exact GPT-5.6 rerun of the 60-row controlled study, so we do not claim that 33%→73% persists under faithful visual feedback.

Question 1. What exactly is the pass condition for the oracle law-fit flag? Is it computed on the evaluator grid and equivalent to the Strict criterion? Please remove this flag and report GPT-5.6 Strict and S/R performance.

Response. We audited both the scorer and the model-visible event stream. The displayed ORACLE LAW FIT field is the Boolean score["ok"], computed on the scene-specific evaluator controls. The model never receives the hidden target values, its JLR, or the evaluator grid itself; it receives only the derived Boolean. In a strict run, this Boolean passes exactly when JLR is at most 0.20. In the paper's relaxed-exponent mode, it is the union of strict predictive fit and the relaxed exponent condition. We agree that this overall Boolean is an avoidable evaluator cue.

We therefore reran the complete 35-scene by 5-setting GPT-5.6 evaluation after removing both the displayed overall field and every prompt reference to it. We preserved the model, hidden-law manifest, action and submission budgets, three target-conditioned execution renders, inverted controls, and per-target physical PASS/FAIL. All 175 flag-off cells completed, and an audit of the environment-generated prompt and feedback stream found no remaining overall law-fit field.

ConditionStrict predictive fitS/RMean JLR
Overall flag visible23.4% (41/175)63.4% (111/175)2.393
Overall flag removed26.9% (47/175)59.4% (104/175)2.594

The paired discordances were 13 visible-only versus 19 removed-only cells for strict predictive fit (exact McNemar p=0.377p=0.377), and 22 visible-only versus 15 removed-only cells for S/R (exact McNemar p=0.324p=0.324). Thus, removing the overall oracle bit does not cause a statistically significant collapse in either binary metric. S/R decreases by 4.0 percentage points and mean JLR becomes modestly worse, so we do not claim that the bit has zero effect; rather, the main result does not depend on exposing it. We will remove the overall field from the revised protocol and retain only the physically grounded target-conditioned renders and per-target PASS/FAIL feedback.

Reviewer_AKikOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether a vision-language agent can discover physical laws from images alone. The authors build a set of simulated physics scenes in which the laws are deliberately changed from the textbook versions, so the agent cannot simply recall them. The agent can add measuring tools to the scene, propose a law, and then test it by checking whether its predictions hold at a few different settings. With a strong model, this agent recovers more laws than existing visual-reasoning agents, and fine-tuning on its successful runs helps a mid-sized open model but not a small one.

Critical strengths
  • The setup is a nice idea: changing the laws from their textbook versions prevents simple recall, and measuring from images tests a step most law-discovery benchmarks skip.
Critical weaknesses
  • The experiments do not clearly support the paper's main claim about the importance of visual instruments. The paper argues that building references like rulers helps agents identify physical laws. However, removing these instruments barely changes accuracy (61.1% vs. 63.4%, p = 0.63). The appendix also notes that instruments mainly reduce token usage rather than improve accuracy. Even without instruments, the agent still outperforms all baselines. This suggests that the improvement comes more from other parts of the agent framework than from the visual instruments themselves.

  • The reported improvement mainly comes from a relaxed evaluation metric, and the tasks are relatively simple. Under the strict metric, the proposed method performs about the same as the best baseline (23.4% vs. 23.8%). Most of the improvement comes from the relaxed metric, which allows fairly large errors in the predicted exponent. Also, each scene only involves one control variable and one observable, and most of the hidden laws are power laws. This makes the task closer to estimating a few parameters than discovering general physical laws. The claims in the title and abstract therefore seem stronger than what the experiments actually demonstrate.

Questions for authors
  1. If instruments do not change accuracy, what explains the gap between your no-instrument agent and the baselines? Were the baselines' prompts and tool interfaces tuned as carefully as yours?
  2. How do the conclusions change if you use strict recovery only, or a tighter relaxed threshold?
  3. Do the findings hold for scenes with more than one control, or for laws that are not power laws?
8 comments

We sincerely thank the reviewers for their time and thoughtful questions, which have helped us clarify and strengthen the paper.

Q1. What explains the gap between the no-instrument OFD agent and the visual baselines? Were the baselines’ prompts and tool interfaces tuned as carefully as OFD’s?

Response. Thank you for this important question. Our system has two complementary components: (i) a visual agent that constructs identifiable evidence from pixels, and (ii) an interaction harness that turns a proposed law into a testable hypothesis. The latter is already useful even when no visual instrument is available, because it supports the loop of changing a control variable, observing the result, submitting a candidate law, receiving counterexamples, and revising the law.

Within OFD, we study progressively stronger feedback contracts:

  • OSBD is a one-shot condition: the agent submits one law and receives no corrective feedback.
  • FGD adds a binary law-fit signal and, after failure, reference renders.
  • GCE-1 executes the agent’s submitted law at one target and returns the resulting render and pass/fail outcome.
  • GCE-n performs the same test at n dispersed targets, making it harder for a law that is only locally correct to pass.

Thus, the no-instrument condition removes the visual-instrument DSL, but it still retains the task-aligned experimental and falsification loop. This explains why “no instrument” does not mean “no useful scaffold.”

We agree, however, that our original comparison did not make the baseline interfaces sufficiently auditable. DeepEyesV2, Visual Sketchpad, and VideoSearcher retained method-specific tool interfaces, and the complete adapters, prompts, and execution traces were not preserved. Moreover, the shared prompt still contained an OFD-specific reference to propose_apparatus, which was not available to these baselines. Although the original evaluation was intended to match the backbone, interaction budget, feedback, and scorer, we cannot verify that the prompt and action interfaces were equally well aligned. We will state this limitation explicitly.

To address it, we rebuilt all three baselines inside a unified Galileo harness. They now share exactly the same:

  • frozen GPT-5.6 backbone and reasoning configuration;
  • scientific-reasoning prompt and five-step analysis procedure;
  • run_experiment, request_more_scenes, and submit_law interfaces;
  • 10-action budget and law-submission limit;
  • three-target execution renders and pass/fail feedback; and
  • termination rules and scorer.

We also replaced the OFD-specific instruction with a method-neutral instruction: use the observation and analysis tools available in the current action space to gather evidence for competing laws. The only remaining difference is the method-specific visual tool:

  • DeepEyesV2 — inspect_region: selects an existing experimental frame, crops or zooms a normalized region, and optionally returns a grayscale or edge visualization.
  • Visual Sketchpad — annotate_observation: draws up to 20 allowlisted lines, boxes, or circles on an existing observation and returns the annotated image.
  • VideoSearcher — search_video_segment: searches a normalized temporal interval of an existing experimental video and returns 1–6 keyframes.

All three tools operate only on pixels already rendered by Galileo. They expose no numerical measurements, hidden laws, additional oracle information, arbitrary code execution, or web search, and every call consumes the same shared action budget.

As an initial behavioral audit, we ran cells covering wave propagation, single-slit diffraction, and resonance. All three tools were correctly registered and available, but the agents chose not to invoke their method-specific visual tool; they used only the shared request_more_scenes and submit_law loop. This supports a plausible explanation for the no-instrument result: generic cropping, annotation, and video retrieval may have limited marginal value for recovering a control–outcome function u↦yu \mapsto y, whereas the task-aligned experiment–falsification–revision loop remains directly useful. It also suggests why domain-specific visual instruments that aims at extracting information through comparison may be easier for an agent to use than general-purpose tools (that focuses more on reading pixels) designed for other visual tasks.

Q2. How do the conclusions change under a tighter relaxed threshold?

Response. Thank you for this important robustness question. We separate the analysis into two aspects: (i) recovery of the law’s functional structure or direction relative to the textbook default, and (ii) absolute calibration of its numerical parameters.

Structural/relative recovery

The paper’s existing relaxed criterion counts a submission as successful if it satisfies the paper’s strict predictive-fit criterion, if the recovered exponent is within an absolute tolerance δ\delta of the hidden exponent, or if the recovered exponent is closer to the hidden exponent than the textbook default. We rescored all 175 archived cells at progressively tighter values of δ\delta.

OFD with instruments

Scoring ruleδ=0.30\delta=0.30δ=0.20\delta=0.20δ=0.10\delta=0.10
Existing structural/relative OR rule63.4%62.9%62.9%

OFD without instruments

Scoring ruleδ=0.30\delta=0.30δ=0.20\delta=0.20δ=0.10\delta=0.10
Existing structural/relative OR rule61.7%61.1%61.1%

Tightening δ\delta has little effect under this criterion: the rates decrease by only 0.5 percentage points with instruments and 0.6 points without instruments. Thus, the paper’s structural/relative conclusion is not driven by the particular choice of δ=0.30\delta=0.30. However, we emphasize that this operational criterion does not by itself establish exact symbolic recovery: different functional forms may remain locally indistinguishable over a finite evaluation range.

Parameter recovery

To isolate absolute parameter calibration, we remove the relative-to-textbook clause and count a submission only when it satisfies the strict predictive-fit criterion or its exponent error is at most δ\delta.

Methodδ=0.30\delta=0.30δ=0.20\delta=0.20δ=0.10\delta=0.10
OFD with instruments48.6%43.4%40.0%
OFD without instruments42.3%41.1%37.7%

Unlike the structural/relative result, absolute parameter recovery decreases materially as the tolerance is tightened. From δ=0.30\delta=0.30 to 0.100.10, the rate falls by 8.6 points with instruments and 4.6 points without instruments. This shows that OFD often identifies the appropriate functional direction or improves upon the textbook default, while precise exponent calibration remains a significant limitation.

Overall, the tighter-threshold analysis supports a narrower conclusion: OFD’s advantage is robust for structural/relative recovery, but not for precise numerical parameter recovery. We will revise the manuscript to separate these two claims and avoid describing finite-grid predictive fit as exact law recovery.

Q3. Do the findings hold for scenes with more than one control, or for laws that are not power laws?

Response. Thank you for raising these two important scope questions. We address them separately.

Laws with more than one control

The original 35-scene benchmark varies one exposed control at a time, so its aggregate results alone do not establish generalization to multiple controls. We therefore added a multi-control hidden-law generalization study with two scene families.

In multi-control free fall, height and gravity are active controls while mass and radius are distractors. In multi-control projectile motion, speed, launch angle, and gravity are active while mass is a distractor. For each family, we generated 20 distinct hidden laws that were not the textbook default. We varied the numerical coefficient and the exponents of the active variables; for projectile motion, we also varied the angular dependence between sin⁡(θ)\sin(\theta) and sin⁡(2θ)\sin(2\theta). We ran one independent episode per hidden law, for 40 episodes in total, using the frozen GPT-5.6 model, a shared 10-action budget, and execution renders with per-target pass/fail feedback. Each final expression was evaluated on 24 held-out multidimensional control combinations.

Hidden-law familyVariable selectionStructural recoveryComplete held-out fit
Multi-control free fall (20 laws)17/20 (85%)6/20 (30%)5/20 (25%)
Multi-control projectile (20 laws)20/20 (100%)4/20 (20%)3/20 (15%)
Overall (40 laws)37/40 (92.5%)10/40 (25.0%)8/40 (20.0%)

Variable selection requires identifying all active controls while excluding distractors. Structural recovery requires recovering the correct multivariate functional form, including the active-variable exponents and angular dependence, while ignoring only an overall multiplicative coefficient. Complete held-out fit requires the submitted expression to achieve JLR≤0.20\mathrm{JLR}\leq0.20 on unseen control combinations.

The results show that the interaction loop transfers strongly to multi-control variable discovery: the agent identified the relevant controls in 37/40 independently parameterized laws. It also recovered previously unseen multivariate structures in 10/40 cases and complete held-out formulas in 8/40 cases. Thus, the agent can discover multi-control laws in a nontrivial fraction of out-of-distribution settings, but structural and numerical extrapolation remain substantially less reliable than identifying which variables matter. We will state this limitation explicitly rather than claiming that the original aggregate findings fully generalize to arbitrary multi-control laws.

This pilot uses the no-instrument action space (run_experiment and submit_law), so it tests transfer of the interaction-and-falsification loop rather than the additional contribution of OFD's instrument DSL. It is reported separately from the original benchmark and is not presented as a paired baseline comparison.

Laws that are not power laws

The original 35-scene benchmark is not restricted to relations of the form y=Cxpy=Cx^p. Ten scenes use non-simple-power functional families, including exponential decay and cooling, a stretched-exponential damping envelope, affine and logarithmic responses, a generalized thin-lens relation, trigonometric refraction and polarization laws, capacitor charging, and resonance. These scenes require the agent to select or construct a functional form rather than merely estimate a power-law exponent.

Successful recoveries do occur in these non-power-law scenes, so the reported findings are not based exclusively on power-law fitting. However, the current aggregate metric combines functional families and does not establish uniform performance across them. We will make this coverage explicit, separate power-law from non-power-law results in the revision, and avoid claiming that all non-power-law families are equally easy.

Comments on the paper

1 comment

Overall Rebuttal Summary

Thank you for the detailed review. We agree that the submission did not clearly separate three claims: the contribution of the experiment–execution–revision scaffold, the contribution of visual instruments, and the auxiliary effect of numeric counterexamples. We therefore audited the evaluation, rescored all archived outputs, removed the evaluator-derived law-fit signal, and added a paired multivariate stress test.

Revised evaluation. The reviewer is correct that the submitted relative-to-textbook clause has no absolute-error cap; we no longer interpret the 63.4% S/R score as exact law recovery. Rescoring all 175 OFD outputs yields 106/175 (60.6%) structural recovery. Among 125 simple-power cells, correct structure plus exponent error at most .30/.20/.10 is 42/125 (33.6%), 41/125 (32.8%), and 37/125 (29.6%). The submitted Strict score, 41/175 (23.4%), is renamed strict predictive fit, since finite-grid JLR does not establish symbolic recovery. Moreover, 26/175 S/R successes rely solely on the relative-to-textbook clause. We will report structure, absolute parameter error, predictive fit, and improvement over the prior separately.

Oracle-signal ablation. We removed the displayed overall law-fit bit and all prompt references to it, while retaining three target-conditioned renders, inverted controls, and per-target PASS/FAIL. All 175 flag-off cells completed. Strict predictive fit changes from 41/175 (23.4%) to 47/175 (26.9%) (p=.377p=.377), and S/R from 111/175 (63.4%) to 104/175 (59.4%) (p=.324p=.324). Thus, neither binary metric collapses significantly without the overall bit, although the lower S/R and higher mean JLR suggest modest calibration help. We will remove this bit from the revised protocol.

New measurement-focused experiment. The original benchmark is dominated by one-control scenes in which the relevant variable is already exposed, making its aggregate ablation a weak test of measurement-dependent discovery. We added a paired 80-law test spanning multivariate free fall, projectile motion, exponential cooling, and rational collision. Each law contains multiple active controls plus a distractor and is evaluated on 24 unseen control combinations. Model, prompt, action budget, feedback, submission schema, and scorer are fixed; only scene-relevant instruments differ. Correct variable sets change from 63/80 to 68/80 (p=.332p=.332), structural recovery from 10/80 (12.5%) to 32/80 (40.0%) (p=1.13×10−4p=1.13\times10^{-4}), and held-out predictive fit from 10/80 (12.5%) to 39/80 (48.8%) (p=8.96×10−6p=8.96\times10^{-6}). Thus, instruments provide significant structural and predictive gains when quantitative measurement is required, though not uniformly across all tasks.

Numeric versus visual feedback. We agree that Table 6 changes both coverage and feedback quantity and does not identify dispersion alone. It is a numeric-feedback, one-revision diagnostic, not the basis of the end-to-end result. We will remove 33.3% to 73.3% from the abstract. A matched-prompt control over 588 Qwen3-VL-32B revisions further shows that faithful renders plus PASS/FAIL yield 3.9% verified laws, versus 16.8% for numeric outcomes; shape error improves in 24% versus 52%. We therefore do not transfer the numeric effect size to the visual loop.

Generality. OFD uses the same model-agnostic interface with proprietary GPT-5.6 and open-weight Qwen3-VL backbones, demonstrating framework portability, not universal performance superiority. Qwen3-VL-8B is an important negative result, and missing matched Qwen3-VL-32B baselines preclude a complete scaling claim. Nevertheless, the new controlled GPT-5.6 experiment establishes significant structural and held-out gains when the backbone can exploit the interaction interface.

In summary, the revised claim is narrower and better supported: Eyes of Galileo provides an executable benchmark for intervention-based visual law discovery; OFD remains effective without the overall evaluator-derived bit; and scene-relevant instruments significantly improve structural and held-out recovery in measurement-dependent multivariate tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.