- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether VLM agents can discover unknown physical laws through active visual experimentation. It introduces Eyes of Galileo, a benchmark of 35 counterfactual physics scenes, and an Observation-Feedback Discovery framework in which agents configure visual measurement tools, propose symbolic laws, and revise them based on execution feedback at multiple targets. The experiments examine the effects of visual instruments, feedback strategies, model capability, and fine-tuning on law recovery. The results show that visual instruments reduce input-token costs, while the benefits of multi-target feedback and fine-tuning vary across model backbones.
Critical strengths
The paper introduces a potentially useful evaluation setting for visual scientific discovery. Agents must configure visual measurement tools, propose symbolic laws, and test them through interventions, making the acquisition of measurement evidence part of the task.
Critical weaknesses
-
Unclear contribution and overly broad framing. The paper combines a benchmark, an agent workflow, and fine-tuning experiments, but does not clearly establish the methodological contribution of OFD over existing interactive law-discovery approaches. The task scope is also limited: each scene focuses on one control–observable relation, and 30 of the 35 scenes use power laws. This provides limited support for broader claims about world-model identification.
-
The relaxed recovery criterion does not ensure predictive accuracy. As written, a predicted exponent passes if it is closer to the counterfactual value than to the textbook value. For example, a prediction of 10 would satisfy this condition when the true exponent is 3 and the textbook exponent is 2, despite being highly inaccurate. The headline 63.4% recovery rate therefore does not necessarily indicate successful law identification; Strict recovery is only 23.4%. The criterion needs clearer justification and implementation details.
-
The central contribution of visual instruments is insufficiently supported. The instrumented agent achieves 23.4% Strict and 63.4% S/R recovery, compared with 25.7% and 61.1% without instruments; neither difference is significant. These results do not establish that constructing measurement references improves law recovery. The token savings are useful, but their attribution to measurement quality is unclear because image sizes also differ between conditions.
-
The role of active experimentation remains unclear. In the instrumented condition, 95.7% of consecutive law submissions involve no additional active observation. Agents also receive an oracle law-fit signal derived from the evaluation grid. The experiments do not sufficiently separate the contribution of active measurement selection from repeated formula revision using this feedback.
-
The feedback experiment does not isolate the claimed mechanism. The improvement from 33.3% to 73.3% comes from a single-revision experiment using numeric feedback. Comparing one target with three simultaneously changes target coverage and feedback quantity. This result cannot be attributed specifically to dispersed counterexamples or directly generalized to the full visual interaction setting.
-
The method’s advantage is inconsistent across models and metrics. With GPT-5.6, the S/R improvement does not translate into superior Strict recovery: 23.4% versus 23.8% for VideoSearcher. With Qwen3-VL-8B, the proposed framework underperforms all three baselines on both recovery metrics. No corresponding baseline comparisons are reported for Qwen3-VL-32B. These results limit the evidence for a general advantage of the proposed workflow.
Questions for authors
-
Clarification of the scoring rule. Does the implemented relaxed-recovery criterion accept the exponent example described above? Are there additional constraints on absolute exponent error, coefficient accuracy, or predictive error? How many reported S/R successes rely solely on the “closer to the counterfactual than to the textbook value” condition?
-
Evidence for the measurement contribution. Given the similar recovery rates with and without instruments, what evidence isolates the contribution of active visual measurement? In particular, are there ablations that remove the oracle law-fit signal while retaining physical-execution feedback?
-
Evidence for the feedback mechanism. Does the benefit of dispersed targets persist with visual feedback and a matched number of targets? A comparison between equally sized sets of concentrated and dispersed targets would help determine whether coverage itself explains the reported improvement.
9 comments
Response to Critical Weakness 2
Critical Weakness 2. The relaxed recovery criterion does not ensure predictive accuracy. As written, a predicted exponent passes if it is closer to the counterfactual value than to the textbook value. For example, a prediction of 10 would satisfy this condition when the true exponent is 3 and the textbook exponent is 2, despite being highly inaccurate. The headline 63.4% recovery rate therefore does not necessarily indicate successful law identification; Strict recovery is only 23.4%. The criterion needs clearer justification and implementation details.
Response. The reviewer is correct about the specific failure case: the implemented relative-to-textbook clause has no absolute-error cap and would accept the stated prediction. We therefore rescored all 175 OFD outputs and replaced the single S/R label with separate measures of structural recovery, absolute parameter recovery, and predictive fit.
| Revised evaluation measure | Result |
|---|---|
| Correct variable/function structure | 106/175 (60.6%) |
| Correct power structure + exponent error | 42/125 (33.6%) |
| Correct power structure + exponent error | 41/125 (32.8%) |
| Correct power structure + exponent error | 37/125 (29.6%) |
| Strict predictive fit (JLR ) | 41/175 (23.4%) |
The 125-cell denominator contains the simple-power scenes and requires both the correct functional structure and the stated absolute exponent tolerance. The 175-cell structural score ignores numerical parameters and requires the correct variables and function family. The former “Strict” score is renamed strict predictive fit, because finite-grid JLR does not establish exact symbolic recovery.
We also decomposed the submitted 63.4% S/R result. The original OR rule counts 111/175 cells. Removing the relative-to-textbook clause, while retaining strict predictive fit or absolute exponent error , reduces this to 85/175 (48.6%). Thus, 26/175 cells rely solely on the relative-to-textbook clause. There is no separate coefficient-accuracy constraint in the relaxed branch; coefficient and full predictive calibration are enforced only through the strict predictive-fit branch.
We will retain the submitted S/R number only for transparency and relabel “closer than textbook” as an improvement-over-prior diagnostic, not law recovery. The revised paper will use structural recovery, absolute parameter recovery, and strict predictive fit as distinct primary outcomes and will no longer describe 63.4% as exact law identification.
Response to Critical Weakness 3
Critical Weakness 3. The central contribution of visual instruments is insufficiently supported. The instrumented agent achieves 23.4% Strict and 63.4% S/R recovery, compared with 25.7% and 61.1% without instruments; neither difference is significant. These results do not establish that constructing measurement references improves law recovery. The token savings are useful, but their attribution to measurement quality is unclear because image sizes also differ between conditions.
Response. We agree only with the narrower observation that the original 35-scene aggregate does not show a uniform recovery gain from instruments. We respectfully disagree with the broader conclusion that constructed measurement references do not improve law recovery. The original benchmark is dominated by one-control scenes in which the relevant variable is already exposed and many laws can be inferred without quantitative measurement, making it a weak test of the instrument hypothesis.
We therefore added a paired 80-law stress test designed specifically to isolate when measurement references should matter. It covers multivariate free fall, projectile motion, exponential cooling, and rational collision dynamics. Every hidden law contains multiple active controls plus a distractor, and every recovered expression is evaluated on 24 unseen multidimensional control combinations. The frozen GPT-5.6 model, prompt scaffold, action budget, physical-execution feedback, submission schema, and scorer are identical between paired arms; the only difference is whether scene-relevant visual measurement instruments are available.
| Outcome | No instruments | Instruments | Paired exact |
|---|---|---|---|
| Correct variable set | 63/80 (78.8%) | 68/80 (85.0%) | .332 |
| Structural recovery | 10/80 (12.5%) | 32/80 (40.0%) | |
| Held-out predictive fit | 10/80 (12.5%) | 39/80 (48.8%) |
The instruments do not significantly change variable selection, but they produce a 3.2-fold increase in structural recovery and a 3.9-fold increase in held-out predictive fit. Both gains are highly significant under paired exact tests. Because the evaluation uses complete symbolic structure and unseen multidimensional combinations, these results cannot be explained by the original relaxed exponent rule or by fitting a one-dimensional evaluation grid.
The effect is task dependent rather than universal: free fall shows little benefit, whereas projectile, cooling, and collision scenes benefit substantially from quantitative references. This heterogeneity explains why the original aggregate, which contains many simple one-control scenes, dilutes the measurement contribution. It also supports a more precise methodological claim: visual instruments are valuable when competing hypotheses require quantitative geometric, temporal, or response measurements; they are not expected to help when the law is already identifiable from the exposed control and raw observation.
The original benchmark also shows an operational efficiency benefit: the instrumented condition uses 23.2% fewer input tokens per successful discovery and exhibits fewer formula-family changes. We agree that image-size differences prevent attributing the entire token reduction solely to measurement quality, so we will describe this as an end-to-end efficiency result rather than a pure causal estimate of reference quality.
Accordingly, we will revise the paper to replace the overly broad claim that instruments uniformly improve all scenes with the supported claim that scene-relevant measurement references significantly improve structural and held-out recovery in measurement-dependent multivariate tasks, while also improving end-to-end efficiency on the original benchmark.
Response to Critical Weakness 1
Critical Weakness 1. Unclear contribution and overly broad framing. The paper combines a benchmark, an agent workflow, and fine-tuning experiments, but does not clearly establish the methodological contribution of OFD over existing interactive law-discovery approaches. The task scope is also limited: each scene focuses on one control–observable relation, and 30 of the 35 scenes use power laws. This provides limited support for broader claims about world-model identification.
Response. We agree that the submission combines several components without separating their roles clearly enough. We will revise the framing around two scoped contributions: (i) Eyes of Galileo, an executable benchmark in which an agent must acquire visual evidence, submit a symbolic law, and test that law through physical interventions; and (ii) OFD, an interaction protocol that exposes task-aligned visual measurement operations and couples proposed laws to target-conditioned physical execution and revision. We do not claim a new general-purpose symbolic-regression algorithm or general world-model identification.
We will also narrow the task-scope claims. The original benchmark varies one exposed control at a time. Our audit identifies 25 simple-power scenes and 10 non-simple families, including exponential decay and cooling, stretched-exponential damping, affine and logarithmic responses, generalized lens, trigonometric refraction and polarization, and resonance. Thus, the benchmark is not limited to 30 power-law scenes, but the reviewer's broader point remains valid: the original benchmark alone does not establish arbitrary multivariate world-model identification.
To test that limitation directly, we added a separate paired 80-law multi-control stress test covering multivariate free fall, projectile motion, exponential cooling, and rational collision dynamics. Every law contains multiple active controls plus a distractor, and complete expressions are evaluated on 24 unseen multidimensional control combinations.
We will report this as a separate stress test, not as proof of general multivariate discovery. The revised claim is that the benchmark and protocol test visually grounded, intervention-based law discovery, with evidence across both single-control and selected multi-control families.