- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EvoSim is an AI scientist framework for physics-based modeling. It revises physical models using experimental feedback and turns successful and failed investigations into reusable modeling procedures. The intended contribution is the joint improvement of the physical model and the agent’s modeling capabilities. Experiments cover lithium-plating onset, dynamic-voltage prediction, and the effect of retained procedures on later modeling tasks.
Critical strengths
The problem is important. Constructing and revising physical models is more demanding than simply running an existing simulator. Checking whether proposed mechanisms are actually implemented and supported by measurements is a useful part of the system.
The results suggest practical value. Across 45 matched pairs, model/physics failures decrease from 33 to 21. This provides encouraging evidence that the learned procedures improve subsequent modeling reliability.
Critical weaknesses
The methodological novelty remains unclear. The paper discusses related experience-learning methods, but the experiments do not establish what Scientific Evolution adds beyond a well-designed procedural memory. In Table 12, Naive Memory omits the exact recurrence and implementation checks that Scientific Evolution receives. The comparison therefore shows that actionable procedures outperform this particular summary, rather than establishing an advantage over a strong experience-learning baseline.
The evidence for continuing co-evolution is also limited. Table 19 documents successive updates, but the 45-pair evaluation uses procedures frozen beforehand. These results support procedure reuse, without isolating the benefit of continued updates. Moreover, the main controlled example concerns implementation repair rather than improved physical-mechanism selection. This leaves a gap between the demonstrated benefit and the broader modeling claim.
Questions for authors
-
Can the authors compare Scientific Evolution with a strong structured-memory baseline built from the same source trajectories, using comparable resources and repeated runs?
-
Does continuing procedure evolution outperform a frozen procedure library under the same budget? Evidence on a task requiring mechanism or equation selection would be particularly helpful.
5 comments
[Part 2/5] Author Response
Weakness 1 (cont.): [...] the experiments do not establish what Scientific Evolution adds beyond a well-designed procedural memory.
Condition for retention. Existing methods retain experience when a task succeeds, or let a language model decide whether to retain it. EvoSim treats each procedure as a testable prediction. A procedure is first tried on other tasks, i.e. tasks that played no part in proposing it. It is retained only if its expected effect actually appears. This is the rule of Section 3.3; we call it the "retention rule" below. DGM also validates changes by experiment, but what it tests is the total score of the whole agent. EvoSim tests each procedure against its own expected effect.
What is updated. Memory methods update only the stored content. DGM modifies the code of the agent but does not accumulate task experience. With the same retention rule EvoSim updates three kinds of content, which the abstract calls knowledge, skills and multi-agent orchestration. Knowledge means physics findings and failure experience. Skills means procedures, tools and checks. Multi-agent orchestration means research strategies and the instructions given to each agent. A research strategy specifies how an investigation is organized: how many candidate branches run in parallel on a task, over how many rounds, and how the final answer is chosen. A "candidate branch" (Section 3.2) is one independent investigation run in parallel on the same task.
[8] and [9] in Table A are two recent preprints that apply skill updating to physics tasks. The skills of HExA also state conditions of use. But both works still retain skills by the pass rate or reward of the task, and update only the skills themselves.
Weakness 2 & Question 1: In Table 12, Naive Memory omits the exact recurrence and implementation checks that Scientific Evolution receives. The comparison therefore shows that actionable procedures outperform this particular summary, rather than establishing an advantage over a strong experience-learning baseline. Can the authors compare Scientific Evolution with a strong structured-memory baseline built from the same source trajectories, using comparable resources and repeated runs?
These two comments are answered by one new experiment. It compares EvoSim with two structured-memory baselines under the same conditions, and removes the three differences above one at a time to see how the result changes. The setup is below; the results are in Part 3.
Baselines. ExpeL [2] stores text insights with importance counts and retrieves similar successful examples. AWM [1] induces step-by-step workflows from successful trajectories. We take AWM as the representative of a well-designed procedural memory.
Task. NewtonBench [3] is a benchmark for physical law discovery. Each task hides a physical law. The law has been deliberately altered, for example in an exponent or a constant, so it cannot be answered by recalling a textbook. The agent collects noisy data through simulated experiments and submits the law it finds. A submission counts as correct only if it is mathematically equivalent to the hidden law. The tasks cover 12 physics domains outside batteries. The law of each domain has 3 altered versions. Each version has 3 observation modes: the output of the law can be observed directly; the law is hidden inside a simple physical system; the law is hidden inside a complex physical system. In the last two modes only the output of the system is observed, so the system must be modeled before the law can be found. We use all tasks of the hard level, 12 × 3 × 3 = 108 in total.
Protocol. There are three steps, the same for the three methods. First, an agent with no experience attempts each of the 108 tasks once and leaves research records. While working on a task the agent does not know whether it is right; after the task ends, each record is labeled correct or wrong. These labeled records are the common source of the three methods. Second, each method builds its own memory from them. The 3 versions of a law are 3 different laws. For a task of one version, a method uses only the records and labels of tasks of the other two versions. The answer to a tested law has therefore not been given to any method before testing. ExpeL and AWM distill as described in their papers. EvoSim builds its procedure library: it proposes procedures from the records, tries them on other tasks, and retains only the procedures whose expected effect appears. Third, testing: each method attempts each of the 108 tasks once with its memory; records made during testing are not labeled. The three methods use the same base model and the same per-task budget. ExpeL and AWM do not update the research strategy; they choose the number of candidate branches themselves, up to a limit. The research strategy that EvoSim uses was obtained by its own updates and is part of the procedure library.
[Part 3/5] Author Response
Weakness 2 & Question 1 (cont.): [...] Can the authors compare Scientific Evolution with a strong structured-memory baseline built from the same source trajectories, using comparable resources and repeated runs?
Table B. Number of tasks on which the correct law is found.
| Row | Method | Direct observation (36) | Simple system (36) | Complex system (36) | Total (108) |
|---|---|---|---|---|---|
| 1 | No experience | 22 | 13 | 9 | 44 |
| 2 | ExpeL | 34 | 19 | 9 | 62 |
| 3 | AWM | 31 | 19 | 10 | 60 |
| 4 | EvoSim, library not modified during testing | 31 | 34 | 21 | 86 |
| 5 | EvoSim, library not modified during testing, w/o retention rule | 28 | 32 | 21 | 81 |
| 6 | EvoSim, library not modified during testing, w/o learned research strategy (single candidate branch) | 29 | 26 | 18 | 73 |
| 7 | EvoSim, library updated during testing | 32 | 33 | 23 | 88 |
Comparison with strong baselines. The memory of ExpeL and AWM does not change during testing. EvoSim in row 4 does not modify its procedure library either, so the conditions are the same. It finds 86 laws; ExpeL finds 62 and AWM 60.
Evidence for the learning signal. When the output of the law can be observed directly, modeling takes few steps, and the three methods are close (EvoSim 31, ExpeL 34, AWM 31). When the law is hidden in a complex system, the system must be modeled first; there are many steps, and looking only at the final outcome discards more. Take the 3 complex-system tasks on Malus's law: all 18 candidate branches fitted the data down to the noise level, and 17 of them had fitted the wrong physical quantity. By final fit error all 17 count as successes, so a method that learns only from success or failure receives no signal here that anything needs correcting. The update of EvoSim reads the test evidence of every candidate model in the research records, not the final outcome. From the records of another domain it obtained a procedure that corrected this error (Example 1 in Part 4). On the 36 complex-system tasks AWM finds 10 laws and ExpeL 9, the same level as the 9 found without experience. EvoSim finds 21, and 23 with the update during testing.
Evidence for the retention condition. Row 5 removes the retention rule: proposed procedures are put into the library without being tried, and everything else is as in row 4. The number of correct laws drops from 86 to 81.
Evidence for what is updated. One of the things EvoSim updates is the research strategy (how the investigation of one task is organized: how many candidate branches run in parallel, over how many rounds, and how the final answer is chosen). Row 4 uses the research strategy that EvoSim learned itself. Row 6 replaces it with the simplest organization: a single candidate branch per task, which receives the whole budget previously shared among the branches; everything else is as in row 4. The number of correct laws drops from 86 to 73. The total budget is unchanged, so these 13 tasks are the gain from the learned research strategy. ExpeL and AWM update only the stored content and have no counterpart to this. The 73 laws of row 6 are still more than those of ExpeL (62) and AWM (60); this part of the gap comes from the first two differences.
Row 7 is the full EvoSim, which also updates the procedure library during testing. It is explained in Part 4.
Repeated runs. In the experiment above, each method attempts 108 different tasks once. In addition, we repeated the experiment of Section 4.3 three times and added ExpeL and AWM. This is the experiment behind Table 12. The task starts from a faulty voltage model: its predicted battery voltage differs from the measurements by about 89 mV. The system has to find the fault and repair the model. ExpeL, AWM and EvoSim use the records of the same earlier investigation. The runs use gpt-5.6-sol on 30 °C DST, with at most 3 solves per run.
Table C. Lowest valid voltage RMSE obtained in each run (mV).
| Condition | Run 1 | Run 2 | Run 3 | Runs repaired |
|---|---|---|---|---|
| Baseline (no experience) | 89.48 | 89.29 | 394.88 | 0/3 |
| Naive Memory | no valid solution | 89.20 | 89.38 | 0/3 |
| AWM | 89.29 | 83.64 | 89.57 | 0/3 |
| ExpeL | no valid solution | 89.38 | 21.56 | 1/3 |
| Scientific Evolution | no valid solution | 21.56 | 21.56 | 2/3 |
Scientific Evolution in the table is EvoSim. It repairs the model in 2 of the 3 runs, reducing the error to 21.56 mV, consistent with the 22.02 mV in the submission. ExpeL repairs it once. AWM, Naive Memory and Baseline never repair it, and their error stays at about 89 mV. The ordering of the methods is the same as in Table B. On this task the procedural memory AWM is no different from the Baseline without experience.
[Part 1/5] Author Response
Thank you for your careful review. Your comments concern four points: the novelty of the method; the comparison with a strong baseline; the role of continued updates; and the evidence on mechanism selection. Parts 1 and 2 state how EvoSim differs from existing methods. Parts 2 and 3 test these differences in a new experiment and compare with strong baselines, including repeated runs. Part 4 addresses continued updates. Part 5 addresses mechanism selection.
EvoSim is not a procedural memory. It differs from a procedural memory in three respects: what signal it learns from; what condition stored experience must meet to be retained; and what is updated besides the stored content. These three points are the method of Sections 3.2 and 3.3 itself. We explain them one by one below, and then test them one by one in an experiment. Scientific Evolution in the submission is the part of EvoSim that accumulates and updates experience; below we write EvoSim throughout.
EvoSim calls each stored practice a procedure. A procedure states four things: what to do; what effect is expected afterwards (expected effect); under what conditions to use it (conditions of use); and which checks confirm it. Besides procedures, EvoSim stores physics findings, failure experience, tools, checks and research strategies. Section 3.1 calls all of this the scientist state, which is what you refer to as the procedure library; we call it the "procedure library" below.
Table A compares EvoSim with existing methods on the three points above.
Table A. EvoSim and existing methods.
The three bold cells are the three differences.
Learning signal. The learning signal of existing methods is sparse: when a task ends there is one outcome, success or failure, and the method learns from that one outcome. In web navigation and games this is reasonable, because tasks are short and the outcome is clear. Scientific modeling deals with frontier problems, unknown mechanisms and complex domains, and aims at models more effective than those of human experts. Such a task is long and hard: one investigation goes through many steps of proposing hypotheses, implementing, solving and testing. Looking only at the final outcome discards the many valuable signals in these steps. The learning signal of EvoSim is dense. Within a task it tests every candidate model, and every test leaves evidence: how far predictions are from measurements; whether the proposed change is actually active in the executed model; whether the result satisfies physical constraints. Rejected candidate models leave evidence too. A model that fits well but does not pass these checks is recorded as a failure in EvoSim.