- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper uses an LLM to propose equations (world models), fits their coefficients numerically and has an independent verifier admit or reject each candidate. When several hypotheses remain, it actively chooses new experiments to tell them apart. Experiments cover a synthetic benchmark (NovelLaw), simulated robot tasks (ManiSkill, RoboCasa), and ODEBench/ODEBase.
Critical strengths
- The pipeline is cleanly designed, with the proposer, fitter, and verifier separated. The agent is sandboxed, and refusals count as failures. The appendix also honestly reports negative results, such as no universal ARS advantage on RoboCasa. The refusal analysis (RQ4) is useful.
Critical weaknesses
- "Discovering laws beyond the initial set" is an overclaim. In NovelLaw, the held-out targets are just new combinations of components already seen during development. For RoboCasa, the grammar in the code allows only bias and velocity plus at most 2 of 11 fixed terms (67 structures), which include exactly the MuJoCo terms. The non-LLM enumeration in the code even does better than the LLM agent.
- Active experimentation does not show a clear benefit, and some evaluations look leaky. On RQ1, ALDER ties random querying (85.0% vs. 85.0%, p=0.203) while using more queries. The RQ2 gain uses a pool deliberately seeded with irrelevant points. In the code, admission requires shift NMSE ≤ 0.02, and the reported shift NMSE seems to come from the same split, which conflicts with Sec. 3.3. For ODE tasks, the agent receives exact derivatives (not stated in A.6.1), while the baselines are copied from other papers rather than rerun. All robot results are in simulation.
- The novelty relative to LLM-based equation discovery and classical model discrimination is unclear. The propose-fit-verify loop is very close to LLM-SR (Shojaee et al., 2025) and SR-Scientist, both cited but not compared. Choosing experiments by disagreement among candidate models is classical model discrimination / query-by-committee (QBC appears only as a baseline copied from prior work on ODEBase). The baselines needed to isolate ALDER's contribution are missing: LLM-SR and SR-Scientist on NovelLaw and the robot tasks with the same data and grammar, PySR/SINDy run on ALDER's actively collected data, an expected-information-gain query selector, and grammar enumeration on RQ3.
Questions for authors
- How do PySR or other LLM-based equation discovery methods perform on NovelLaw with the same data and grammar? How does plain enumeration do on RQ3?
- Is the reported NovelLaw shift NMSE computed on a split that is not used for admission?
- Did the ODE baselines receive exact derivatives? What happens with noisy, numerically differentiated data?
- How do LLM-SR and SR-Scientist perform on NovelLaw and RoboCasa under the same data and interaction budget?
5 comments
Equation Search versus Active Observation Acquisition
Thank you for encouraging direct comparisons with LLM-SR and SR-Scientist under shared observations and interaction budgets. Your question helps us clarify an important distinction between iterative equation search and active observation acquisition.
The standard workflows of LLM-SR and SR-Scientist operate on supplied observations. LLM-SR improves equations through candidate search and numerical fitting, while SR-Scientist uses data-analysis tools, equation evaluation, and iterative feedback. Neither standard workflow includes an acquisition mechanism that autonomously selects and executes environmental interventions to obtain new observations. SR-Scientist’s autonomous tool use concerns analysis and equation refinement on available data, rather than active acquisition of environmental evidence.
ALDER connects equation discovery with active observation acquisition: disagreement among candidate models can guide the next intervention, and the resulting observations support further equation revision.
Accordingly, in our reported NovelLaw shared-observation comparison, we provide both methods with the public observations acquired by ALDER, allowing their respective equation-search procedures to operate on identical evidence through the described adapters. This evaluates equation discovery from shared observations; it does not attribute ALDER’s intervention choices to either baseline or compare independent acquisition policies.
Additional RoboCasa Results: Comparison with LLM-SR and SR-Scientist under Matched Observations
We thank the reviewer for this constructive suggestion, which motivated us to strengthen the direct comparison with LLM-SR and SR-Scientist. Building on our earlier NovelLaw experiments, we have now completed the comparison across all 80 RoboCasa environments, comprising 40 Drawer and 40 Door environments.
Both baselines receive exactly the same public observations acquired by ALDER, while retaining their respective search and numerical fitting procedures. ALDER’s candidate equations, fitted coefficients, and protected validation feedback are not provided to the baselines. All three methods use a common LLM backbone, with at most 12 LLM calls per environment across the entire discovery process. All new baseline models are frozen before evaluation on the same independent test trajectories. Recovery requires force-prediction NMSE ≤ 0.03.
| Method | Drawer | Door | Overall recovery |
|---|---|---|---|
| ALDER | 38/40 | 40/40 | 78/80 (97.5%) |
| LLM-SR | 37/40 | 29/40 | 66/80 (82.5%) |
| SR-Scientist | 39/40 | 26/40 | 65/80 (81.25%) |
With identical observations and the same call cap, ALDER achieves 97.5% overall recovery, exceeding LLM-SR and SR-Scientist by 15 and 16.25 percentage points, respectively. The improvement is particularly clear on Door, where ALDER recovers all 40 environments, compared with 29 for LLM-SR and 26 for SR-Scientist. SR-Scientist also performs strongly on Drawer, recovering 39/40 environments.
We appreciate the reviewer’s recommendation: this additional comparison strengthens the empirical evidence for ALDER’s ability to recover predictive equations from limited robotic observations and clarifies its performance relative to existing LLM-based discovery methods.
Thank you for your thoughtful review and for highlighting the separation of proposal, fitting, and verification, the sandbox, and the refusal analysis. ALDER integrates revisable proposals, evidence acquisition, protected admission, and downstream use within a supplied grammar. NovelLaw tests new compositions of available components. RoboCasa uses physical features in simulation. We build on disagreement-based model discrimination.
1. NovelLaw baselines and RoboCasa enumeration
On all 60 NovelLaw cases, baselines received the public fitting and development observations acquired by ALDER. Hyperparameter and program selection used public data. We accessed test points only after fixing models.
Recovery requires test NMSE ≤ 0.02 and counts failures. We report the declared configurations and retain both PySR operator settings and shorter LLM runs. SINDy uses a fixed library, PySR searches over operators, ALDER uses a typed AST, and the LLM adapters use their own program interfaces. We document each search budget. The frozen ALDER reference retains its proposal and admission history.
In RoboCasa, enumeration of the 67 allowed structures with ALDER’s acquired data and public-validation selection admitted 38/40 Drawer and 14/40 Door, versus 40/40 each for frozen ALDER. Other selection variants match ALDER’s Drawer coverage. With robot evidence fixed, admissions in Drawer/Door were 35/40 and 31/40 with the original grammar, 28/40 and 20/40 without the linear position term, 18/40 and 13/40 without both friction-sign terms, and 37/40 and 30/40 with ten added terms. These ablations measure sensitivity to the supplied grammar.
2. Independent NovelLaw Shift evaluation and acquisition
We used the original Shift split for admission and will label its scores as admission-set diagnostics. We evaluated the frozen final models on 4,096 new points per case in each domain, reserved for final testing. On the same 60 target equations, ALDER recovered 50/60 on independent Shift (median NMSE 0.00630) and 51/60 on the broader functional domain. We retained all cases and failures.
In 60 NovelLaw cases (three paired runs each), ARS and random shared initial proposals, fitted coefficients, and budgets. Functional recovery was 97/180 vs 95/180. In the observation-use comparison, mean extra points were 3.33 vs 3.73 per run (10.7% fewer for ARS).
With online stopping, ARS vs random identified 195/200 vs 189/200 on uniform pools and 194/200 vs 186/200 on Latin hypercube pools, averaging 1.085 vs 1.735 and 1.065 vs 1.830 queries, respectively. Monte Carlo expected information gain identified 195/200 on both pools, averaging 1.110 and 1.135 queries.
3. ODE derivatives and observation sensitivity
The original ALDER runs used exact derivatives. The literature baseline scores alone leave derivative access unclear. We reran full ALDER and baselines on 12 prespecified systems stratified by dataset and dimension. With exact derivatives, shared trajectory evaluation, and blinded symbolic scoring, functional recovery was 11/12 ALDER, 6/12 LLM-SR, and 8/12 SR-Scientist. We report all 14 methods or variants, budgets, and failures. Passive methods received ALDER’s final public data. Active methods shared a 20-point start and could query up to two more 20-point trajectories.
A separate fixed-data study used the same systems and trajectory evaluator.
Differentiation used fixed
numpy.gradient. All models were frozen before testing. Recovery requires trajectory NMSE ≤ 10⁻⁶ in reconstruction, generalization to unseen initial conditions, and contiguous future extrapolation.4. LLM-SR and SR-Scientist with shared observations
The comparison requested here was completed on NovelLaw only. On the independent Shift test, moving from the same passive observations to ALDER’s final public observations raised LLM-SR recovery from 8/60 to 21/60 and SR-Scientist from 21/60 to 36/60. Their acquired data came from ALDER-selected interventions.
The 12-call LLM-SR adapter retains the pinned official island sampler and buffer. The 12-turn SR-Scientist adapter retains the official prompts, tools, and history interface. The adapters share backend and reasoning settings. We document grammar, program size, and fitting adaptations, along with the ALDER reference’s historical settings.