acceptodds
Under review as a conference paper at ICLR 2027

Discover-Many-Worlds: Reinforcement Learning for Scientific Discovery in Automatically Generated Physics

Abstract

Scientific discovery in physics requires a process of designing experiments, revising hypotheses, and inferring unfamiliar laws from observations. We introduce Discover-Many-Worlds, a reinforcement learning framework for training scientific discovery agents in automatically generated physics worlds. Our world-generation pipeline combines LLM-driven construction, procedural simulation, and verification testing to generate and validate diverse environments at various difficulty levels with no human intervention. We train a 4B-parameter Qwen model with GRPO, increasing its capabilities on DiscoverPhysics up to the performance level of Claude Sonnet 4.6. We also observe improved performance on other scientific and reasoning benchmarks. We demonstrate that the RL-trained agent engages in more extensive experimentation, where it gathers more observations before committing to an explanation and considers a broader range of physical laws instead of defaulting to familiar interactions. These results demonstrate that automatically generated physical worlds can provide effective training environments for experimentation and reasoning about physical phenomena, both of which are essential for scientific discovery.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Reinforcement learning, automatic training data generation, scientific discovery, agentic discovery, scientific reasoning, LLM agents, AI for science

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TU89OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission proposes a pipeline for generating new variations of environments (worlds) in the format of the DiscoverPhysics (Wiemann et al., 2026) framework.

The 'easy' worlds are constructed by varying several parameters of the DiscoverPhysics force laws. The 'hard' worlds are generated by prompting a frontier LLM (Claude Fable 5) to produce worlds in the style of DiscoverPhysics with some latent structure governing the dynamics; the prompt (given in full in Appendix F) instructs that each new world "must differ substantially" from the previous ones.

Then, the medium-difficulty worlds are generated using a simplified version of the above prompt, and kept if a base Qwen3-4B LLM attains an evaluation score in [0.4, 0.7]. For worlds outside this range, a frontier LLM (Claude Opus) receives a prompt containing the world's score and a "too hard" or "too easy" label, and revises the world or writes a new one. The resulting world is then tested with Claude Haiku 4.5 instead of Qwen3-4B, and kept if Haiku scores in [0.4, 0.7].

Finally, using the verl library, Qwen3-4B is fine-tuned with GRPO on the resulting worlds. The fine-tuned model is then evaluated on the hand-made DiscoverPhysics worlds, where it outperforms Claude Sonnet 4.6 on most metrics from the DiscoverPhysics work.

Critical strengths
  • Fine-tuned with GRPO, Qwen3-4B outperforms Claude Sonnet 4.6 on most DiscoverPhysics metrics when tested on the original DiscoverPhysics worlds. Fine-tuning removes most execution errors, such as crashing code and premature submissions, as shown in Figure 4.

  • The medium worlds are calibrated to a target difficulty: a frontier LLM (Claude Opus) revises or replaces each world using its measured score until a reference model scores it in a target band. This loop extends ideas from previous works, such as unsupervised environment design, to the problem of discovery of physics rules and provides a reusable way to fill the gap between easy and hard worlds.

  • 'Hard' worlds are reimplemented by an LLM from their written description alone, with an adversarial test for cases where the two versions diverge. Unlike LLM environment generators such as OMNI-EPIC (Faldor et al., 2025), this is designed to reject worlds whose description underspecifies their code or whose numerics are unstable, and it gives others a reusable test for LLM-written simulators.

Critical weaknesses
  • Transfer beyond DiscoverPhysics is mixed, so the abstract's claim of improved performance on other scientific and reasoning benchmarks holds only in part. NewtonBench, which the paper ranks most similar to its training task, and HeroBench improve, but GravityBench is "slightly degraded", and GravityBench, HeroBench and LiveCodeBench scores peak at step 500 and fall by step 850 (Figure 5). Transfer is therefore benchmark-dependent, and for most benchmarks it does not grow with further training.

  • The easy training worlds overlap with the DiscoverPhysics test worlds, which the paper calls an "in-distribution test set": several easy worlds use the same force laws as public test worlds, differing only by a constant factor. So a large part of the benchmark prediction-error gain may come from familiar force laws rather than new discovery.

  • The results show that the full pipeline can help, but there is a lack of a systematic analysis of why various parts of the pipeline work and to what extent each of them is needed.

Overall, the submission appears to be a reasonable training setup, though perhaps not a significant advance. Its training loop combines existing parts, the DiscoverPhysics simulator and protocol and standard GRPO in the verl library. Its own contribution is the world-generation pipeline, whose value the experiments do not isolate sufficiently well.

Questions for authors
  1. How much of the headline drop in prediction error comes from a few worlds such as gravity? The largest gain in Figure 6 is on gravity, whose force law also appears in the easy training worlds, yet coulomb_easy, whose force law also appears there, gains only modestly, and several public and private worlds barely change. These gains seem far from the "uniform improvement to the normalised MSE for every world" that the main text reports.

1.1. Why is Figure 6 placed only in Appendix C, when Section 5.2 draws its per-world conclusions from it? Readers cannot easily check the main-text claims without leaving the main paper.

  1. Does the higher explanation score, one of the metrics on which fine-tuned Qwen3-4B beats Claude Sonnet 4.6, reflect better understanding or just wording that the scoring LLM favours? It is not easy to untuit this from the information I could see in the paper so far, but apologies if I missed something.
2 comments
AuthorsOpenReview ↗

Response and extra ablations

Firstly, we do agree with the reviewer that the cross-benchmark improvements are modest in most cases. We would like to point out that we believe it is of interest to the community to show if improving scientific discovery ability (as measured by DiscoverPhysics) correlates with improvements on other tasks.

Q1: We also agree the importance of the LLM-generated worlds was not adequately shown, hence we performed a direct comparison to the same Qwen3-4B model trained on only the template worlds, showing the results below.

Metric (22 worlds × 5 seeds)Template only (78 worlds)Full pool (127 worlds)Template − Full [95% CI]
Normalised MSE, geometric mean (↓)0.590.085×6.9 worse [1.5, 49]
Fit pass rate (MSE < 0.1)0.170.35−0.18 [−0.35, −0.04]
Explanation score0.320.34−0.03 [−0.07, +0.01]
Both-axes pass rate0.000.05−0.05 [−0.14, 0.00]
Shaped reward0.290.41−0.12 [−0.21, −0.04]

Figure here: https://postimg.cc/Wt60MwDd

If the gains came from familiar force laws, training only on the 78 template worlds, which contain all of those force laws, should reproduce them. It does not according to the geometric-mean MSE is 0.59 vs 0.085 for the full pool, mirrored in the fit pass rate halving, and the template-only run plateaus without reaching the pass threshold. The automated generation pipeline is therefore the component that drives the trajectory-accuracy gain, even for an identical underlying model. We show a breakdown of the performance on the worlds in the evaluation set that crossover with template worlds, compared to those that do not in the figure below. We see gains across all world types, especially when using the full template + self-generated world training set.

Figure here: https://postimg.cc/xcdZbCsn

We also point to the analysis done in the response to Q3 of Reviewer fxys for further evidence that performance increases on DiscoverPhysics worlds that are not similar to the laws seen in the template worlds.

Q1.1: We also agree that Fig. 6 is useful in the claims for Section 5.2, and we will move the figure there from the appendix accordingly. This makes it more clear that we see gains in the MSE fit to 21/22 worlds, which we also further clarify in the text.

Q2: To test this, we perform a judge ablation where we rescore each world’s explanation with three non-Claude judges. We see the three alternate LLM-judges give improvements to the explanation score of equal to, or more than the improvement on the Claude Sonnet judge, so we do believe the RL training has resulted in greater conceptual understanding of these worlds.

JudgeStep 0 (base)Step 850Gain
Claude Sonnet 4.6 (original)0.190.33+0.14
Claude Sonnet 4.6 (re-run)0.190.33+0.14
GLM-5.20.250.40+0.15
MiniMax-M3.1 Flash0.280.39+0.11
DeepSeek-V4 Pro0.210.35+0.14

Figure here: https://postimg.cc/BPqLhR8Q

We also do a “key word” sensitivity test where we scored explanations for 4 benchmark worlds (yukawa, extra_dimensions, coulomb_easy, and one private world) under four conditions, with all four judges and 3 samples per judge:

ConditionClaude Sonnet 4.6GLM-5.2MiniMax-M3.1 FlashDeepSeek-V4 Pro
A. Canonical names, wrong physics0.13 ± 0.050.26 ± 0.090.23 ± 0.070.18 ± 0.08
B. Correct physics, no canonical names1.00 ± 0.001.00 ± 0.000.99 ± 0.031.00 ± 0.00
C. Correct physics, canonical names1.00 ± 0.001.00 ± 0.000.99 ± 0.031.00 ± 0.00
D. Failed policy explanations0.20 ± 0.140.29 ± 0.130.29 ± 0.090.23 ± 0.15
D + appended keyword sentence0.24 ± 0.120.33 ± 0.130.39 ± 0.090.30 ± 0.14

Every judge scores the correct-but-unnamed physics (B) a near perfect score, and the named-but-wrong physics (A) near the bottom well below a pass. So the score is driven by explanation accuracy rather than terminology. Appending a sentence of rubric keywords to real failed explanations (D) gives limited partial credit (+0.04 to +0.10), and padded explanations remain far below correct ones.

Thank you for your response and for reporting the results of additional experiments. After taking these experimental results into account, I still have concerns about some of the central claims.

The messaging in the title and abstract implies that training on the generated worlds improves scientific discovery ability. The Conclusion states: "the RL training improved the scientific discovery ability of the Qwen model"; the Abstract states that the generated worlds "can provide effective training environments for experimentation and reasoning about physical phenomena" and claims "improved performance on other scientific and reasoning benchmarks."

The main results on DiscoverPhysics use an in-distribution test set, and the cross-benchmark gains are modest. Of the 11 public worlds, 7 overlap with the training templates. In the response to Reviewer fxys, the results on the remaining 4 non-overlapping worlds are now reported separately. While the results improve over the baseline in terms of trajectory fitting, a "pass" requires both low MSE and an explanation score of at least 0.75. None of the 20 attempts pass, and the mean explanation score only reaches 0.28. So, while the experiments show improvement over the baseline, they do not rise high enough so that one could claim successful discovery of unfamiliar laws. Claims about "Scientific Discovery" would need support from strong performance in out-of-distribution cases. Otherwise, the improvement can be attributed to better trajectory fitting, not to law discovery.

Reviewer_fxysOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper constructs a fixed pool of 127 physics worlds: 78 template-based easy worlds, 22 medium worlds generated and calibrated with language models, and 27 hard worlds with hidden structure. A Qwen3-4B-Thinking agent runs experiments, fits candidate laws, and submits code and an explanation. GRPO training combines a held-out trajectory reward with an LLM-judged explanation score. After 850 updates, the agent improves on the 22-world DiscoverPhysics benchmark, especially in trajectory prediction and executable code.

Critical strengths
  1. The paper introduces an interactive training environment for scientific research. Agents can independently design experiments, observe the results, and gradually refine physical laws. This allows models to learn a process closer to real scientific research, rather than simply predicting equations from fixed data.

  2. The training shows clear improvements, with a useful analysis of failure cases. After multi-turn reinforcement learning, the small model achieves lower trajectory prediction errors and higher scientific explanation scores. Each DiscoverPhysics world is evaluated with five independent runs. In particular, the number of runtime crashes is significantly reduced, suggesting that training improves the model's ability to execute experiments reliably.

Critical weaknesses
  1. The paper lacks key ablation studies, making it unclear what drives the performance gains. It does not compare template-only training with additional medium and hard worlds, or trajectory-only rewards with the added 30% explanation reward. Section 5.1 also shows that the model generates about three times more tokens and uses more interaction rounds after training. Thus, the gains may come from better scientific reasoning, improved tool use, or simply more effective use of the interaction budget.

  2. The reliability of the training environments and the model's generalization ability are not fully verified. Section 3.3 only checks numerical stability for medium worlds, which does not guarantee consistency between physical laws, task descriptions, and evaluation criteria. More importantly, the main benchmark is in-distribution and shares similar simulators and physical backgrounds with the training data. Without testing on different physical laws or task types, the results are insufficient to demonstrate generalizable scientific discovery abilities.

Questions for authors

Q1. Under the same training compute budget, how does model performance change when using only template-generated worlds or only trajectory prediction rewards?

Q2. Does the trained model still show significant improvements over the base model when both are given the same token, experiment, and fitting-tool call budgets?

Q3. Which physical laws overlap between the training and evaluation sets? Can the authors report performance on genuinely new physical mechanisms or test on different families of physical laws, rather than just changing parameters or system structures?

Q4. For the 22 medium-difficulty worlds, how do the authors verify the correctness of task descriptions and evaluation criteria? How do they ensure that agents can actually discover the hidden physical laws through experiments?

2 comments
AuthorsOpenReview ↗

Response to questions 1 and 2

We thank the reviewer for highlighting the similarities of our interactive training environment to the real scientific research process and the improvements from training in achieving lower errors in trajectory prediction, higher scientific explanation scores, and reduced numbers of runtime crashes, leading to the model’s increased ability to execute experiments reliably.

In response to the weaknesses, we perform two additional ablations: a training run with the easy-only template worlds, and a training run with the reward function only comprising the standard MSE and a term from computing the adversarial MSE (instead of the judge’s explanation score), making the training only dependent on the model’s MSE fitting and trajectory prediction. We also would like to point out the paper’s Benchmark Transfer section (5.3) which does evaluate the model on the transfer to out-of-distribution tasks, including coding tasks and other physics laws not found in the training data or the DiscoverPhysics evaluation.

Q1: We trained the same model, Qwen3 4B with a modified reward function where the model only gets trajectory prediction rewards. We leave the MSE portion of the reward function the same, and we replace the judge evaluation score portion of the reward function with the adversarial MSE (as described in the LLM-generated world section). The test cases for the adversarial MSE are found by choosing the three worst test cases from an adversarial search over the initial conditions for the particle positions, velocities, and measurement times. We train for 850 steps on the same worlds as the original RL run described in the paper, with the same token budget (65,536-token budget per episode) and same number of GPUs (64 H200s). We see comparable results on the MSE pass rate as with the original run, but this ablation suffered from worse evaluation scores. Here, it seems that the model needs a reward signal that incorporates the explanation score in order to perform well on explainability, not just simply curve-fitting.

Figure here: https://postimg.cc/GTdZfxJ9

Metric (22 worlds × 5 seeds)Trajectory reward only (127 worlds)MSE + judge reward (127 worlds)Trajectory only − MSE + judge [95% CI]
Normalised MSE, geometric mean (↓)0.0680.085x1.2 better [0.6, 2.8]
Fit pass rate0.390.35+0.04 [−0.09, +0.18]
Explanation score0.280.34−0.06 [−0.12, −0.01]
Both-axes pass rate0.030.05−0.02 [−0.05, 0.00]
Shaped reward0.400.410.00 [−0.08, +0.09]

We point the reviewer to the ablation of training on easy-only template worlds in the response to Reviewer 29GK.

Q2: All our comparisons between the base model and the RL-trained model use the exact same token, experiment, and fitting-tool call budgets. The model always gets 1 fitting-tool call per experimental round, with 10 experimental rounds in the budget. So Figures 2, 3, 4, 5, and 6, for example, follow these rules, and we see gains in performance over the base model.

AuthorsOpenReview ↗

Response (cont.) to questions 3 and 4

Q3: Of the 11 public benchmark worlds, 7 are similar to an easy, template training world with the same law up to constants, particle count, or which particle in the world is static. These 7 are gravity, coulomb_easy, fractional, circle, hubble, ether, three_species. However, the increase in performance from training is not concentrated in just those worlds, as the MSE pass rate (norm_MSE < 0.1) and explanation score increases on worlds in both groups:

Public benchmark worldsWorldsMSE pass, step 0MSE pass, step 850Explanation, step 0Explanation, step 850Full pass, step 850
Law matches an easy template70/35 (0%)15/35 (43%)0.270.495/35 (all gravity)
No easy-template law42/20 (10%)10/20 (50%)0.120.280/20

We would like to point out that the gravity world is very similar to some of the template worlds, corresponding to the high pass rate. However, other benchmark worlds have no analogue in the training data, neither in the template worlds nor the medium or hard LLM-generated worlds. For example, the yukawa world has no world similar to its mechanism in the RL training data, yet performance increases on it after training.

Additionally, we see the improved scientific capabilities in the thinking traces of the trained model at step 850 as compared to the base model. Here we show actual excerpts from the model before training and after. For the gravity world, the model still experiments with inputs and then performs a fit instead of just recalling the worlds it saw in RL training.

gravity, step 850 (episode 0: norm_MSE 6x10e-9, explanation 0.9):

The model varies one input at a time, then fits the constant.

"So, when r doubles from 5 to 10, the acceleration went from ~0.0318 to ~0.016. Let's compute the ratio: 0.016 / 0.0318 ≈ 0.5. So it's roughly halved, not quartered. If it were inverse square, it should be 1/4, but it's about half."

"The measured a was ~0.0318, so k/5 ≈ 0.0318 → k ≈ 0.159."

The fitting tool returns k = 0.15915, the benchmark's 1/(2π). The templates use 1 and 2 for k.

gravity, step 0 (episode 2: norm_MSE 0.92, explanation 0.5):

"This is a different law than inverse-square (which is 1/r²), but matches the observed scaling (1/r). Next, I'll run an experiment with p1=2.0 to see if the acceleration scales with p1."

It never runs that experiment and submits a = p1 * p2 / r**2.

Similarly, for yukawa we see improvements in the model’s scientific reasoning and experimentation, where it actually measures parameters instead of simply recalling from pretraining data or guessing.

yukawa, step 850 (episode 2: norm_MSE 0.037, explanation 0.3):

This world's screened force has no template law. Over r = 5–10 it falls like r^−4.2. The model's accelerations (0.0058 at r = 5, ~0.0004 at r = 10) match the true values (0.0059, 0.0003), and it solves for the exponent:

"14.5 * 5^n = 10^n … 14.5 = 2^n → n = log2(14.5) ≈ 3.86, which matches earlier. So n≈4."

"0.0058 = k * 1 / 5^4 → k = 0.0058 * 625 ≈ 3.625."

The template exponents run from −3 to +2, so this is measured, not recalled.

yukawa, step 0 (episode 0: norm_MSE 0.81, explanation 0.3):

It started the particle at r = 1 and r = 2. Its own velocities imply an exponent near 1.5, but it computes no exponent:

"This suggests that the acceleration is weaker at larger distances."

"…the acceleration is stronger when the particle is closer (r smaller), which matches inverse-square."

It submits 1/r^2 with the constant left at its default of 1.0.

Q4: Medium worlds were checked by the numerical robustness check, ensuring that any world that was improperly specified in code was rejected. They were also checked by running a discovery agent on all of them for difficulty calibration, Claude Haiku (16 rounds, 3 attempts, resulting in median explanation scores of 0.4-0.7 on every world), which shows that their physics is recoverable through experiments. The final trained model passed 7 of these 22 worlds, and on 16 of the worlds the model passed the MSE portion of the passing requirement. To ensure that the worlds do have discoverable laws, we run the reimplementation check that the hard worlds underwent on the medium worlds as well. This check gives an agent just the description of the world, and it must reimplement the Python code implementation, which is then checked via MSE against the original. All the hard worlds undergo this check. All 22 medium worlds passed reimplementation, meaning that their physics was described well enough to be reproducible in code. Here we show the reimplementation results compared to the results listed in the paper for the hard worlds to show the efficacy.

Medium worlds (new)Hard worlds (paper)
Reimplementation from description22/22 pass; MSE median 8e-21, max 9e-1827/27
Adversarial disagreement, median / max1.2e-18 / 8.7e-91.5e-15 / 7.9e-5
Disagreements found00
Reviewer_29GKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the lack of training environments for RL-based scientific discovery by automating the generation of physics worlds compatible with the DiscoverPhysics protocol. The pipeline produces 127 worlds: 78 easy template worlds (parameter variations of 3 of the 8 DiscoverPhysics force-law kernels, no LLM), 22 medium LLM-generated worlds (refined until a reference agent scores in a target band), and 27 hard worlds built around hidden latent structure.

Critical strengths
  • The problem framing is excellent, scientific discovery lacks a Lean-style verifier.
  • The behavioral and failure-mode analyses convincingly show what changed.
Critical weaknesses
  • 78 of 127 training worlds are template-based, and training uses only two-particle medium worlds, so the paper cannot show that the LLM-generated medium/hard worlds—the actual contribution—are responsible for any of the gain versus the cheap template worlds alone.
  • Since the judge is presumably a Claude model while the generators are Claude models and the training reward optimizes against it, family bias and judge-gaming risks need to be addressed (e.g., judge agreement with human ratings on a sample).
Questions for authors

I would raise my score if the rebuttal provides the template-only ablation, identifies and validates the judge, and ablates the reward and difficulty calibration.

1 comment
AuthorsOpenReview ↗

Response and ablation test

We thank the reviewer for the comments and positive review of the paper, and agree these ablations are important to emphasise the importance of both the LLM-generated worlds, and show the conceptual improvement does not just come from overfitting to specific judges.

We perform an ablation over 3 judges specifically GLM-5.2, MiniMax-M3.1 Flash, and DeepSeek-V4 Pro, as well as run the original Sonnet 4.6 judge again. We see significant improvement in the explanation scores between the base (white) and RL trained Qwen3.5-4B model across all judge families, with all judges scoring the mean explanation score at least 0.11 points higher (out of 1) after the training, which is equal to or higher than the improvement seen in the Sonnet 4.6 judge.

JudgeStep 0 (base)Step 850Gain
Claude Sonnet 4.6 (original)0.190.33+0.14
Claude Sonnet 4.6 (re-run)0.190.33+0.14
GLM-5.20.250.40+0.15
MiniMax-M3.1 Flash0.280.39+0.11
DeepSeek-V4 Pro0.210.35+0.14

Figure here: https://postimg.cc/BPqLhR8Q

We also present a comparison between training the same Qwen3-4B model on the template worlds only (the easy worlds in the paper's definition). While the judge scores are similar, there is significant degradation to the MSE score, with none of the worlds passing the 0.1 threshold when training only on the template worlds.

Metric (22 worlds × 5 seeds)Template only (78 worlds)Full pool (127 worlds)Template − Full [95% CI]
Normalised MSE, geometric mean (↓)0.590.085×6.9 worse [1.5, 49]
Fit pass rate (MSE < 0.1)0.170.35−0.18 [−0.35, −0.04]
Explanation score0.320.34−0.03 [−0.07, +0.01]
Both-axes pass rate0.000.05−0.05 [−0.14, 0.00]
Shaped reward0.290.41−0.12 [−0.21, −0.04]

Figure here: https://postimg.cc/Wt60MwDd

We also show below in blue the template-only training compared to the full training (orange) for each world in the evaluation set. The left column shows the mean normalised MSE with the right showing the explanation scores.

Figure here: https://postimg.cc/5XjvWcsz

We also show a breakdown of the performance on the worlds in the evaluation set that crossover with template worlds, compared to those that do not in the figure below. We see gains across all world types, especially when using the full template + self-generated world training set showing the addition of the non-template worlds help both the MSE fitting, and explanation of the template worlds more than training on template worlds alone.

Figure here: https://postimg.cc/xcdZbCsn

Comments on the paper

1 comment
AuthorsOpenReview ↗

New Qwen3.8 27B results

We thank all three reviewers for their careful reviews, and we are glad they found the problem framing compelling and the training pipeline a useful contribution. We also thank the reviewers for their helpful comments, and we address the weakness and questions brought up with ablations and clarifications. These changes are also reflected in the pdf.

We would like to highlight a new result since submission showing the applicability of this training pipeline to other models. We have trained a Qwen3.8 27B model using a slightly modified version of this pipeline, with a new pool of generated world templates built with stricter validation checks the reviewers asked about. We have improved the performance of the 27B model from a pass-rate of 13.6% to 32.7% on DiscoverPhysics with no noise and from 7.3% to 19.1% with noise. We train using the same GRPO pipeline and same explanation judge (Claude Sonnet 4.6), but with this expanded set of automatically-generated worlds.

In particular, we have modified all worlds to be the form of a template with the generator model specifying a range of values valid for each of the parameters in the generated force law. This allows us to sample individual worlds during training with unique parameter values for each force law mechanism, expanding the training data to almost infinite worlds and so that no world is exactly repeated. The easy worlds are still modified versions of the force law kernels built into the DiscoverPhysics code, but now the parameters are sampled instead of chosen a priori by us. The medium and hard worlds are generated by the model. One other change to simplify the refinement and validation process was to allow the generator model to test its ideas using Python and bash. This allowed the model to design more robust laws through experimentation, reducing the number of refinement and validation checks and rewrites. We used Claude Opus 5.5 as the generator agent within the default Claude Code harness.

Here we show results as a proof-of-concept. We only had enough computational resources to train for a short time with two configurations of hyperparameters, but it is possible that increased performance gains could be achieved with different hyperparameters or training setups.

ModelTraining checkpointsObservation noiseFull passnorm-MSE passMean explanation score
Qwen3-4B-Thinking (paper)step 0 → 850none0.0% → 4.5%2.7% → 35.5%0.19 → 0.34
Qwen3.8-27B run 1step 0 → 125none13.6% → 32.7%45.5% → 60.0%0.42 → 0.56
Qwen3.8-27B run 2step 0 → 755%7.3% → 19.1%39.1% → 54.5%0.35 → 0.47

In run 1, DiscoverPhysics worlds with at least one full pass go from 6 to 11 (out of 22), and the number of full passes increased on 9 worlds. The remaining worlds did not decrease or increase in the number of passes. The largest gains are (out of 5 attempts): hubble (1 to 5), oscillator (2 to 5), ether (0 to 3), coulomb_easy and yukawa (1 to 3). In run 2, the number of full passes increased on 7 worlds and stayed the same on the rest.

Run 1 has the same RL setup as Qwen3 4B in the paper: noise-free observations, reward=0.7 x MSE term + 0.3 x judge, 32 worlds x 8 episodes per step. Run 2 adds 5% observation noise in training and evaluation, reward=0.5 x MSE term + 0.5 x judge, and 16 (instead of 8) episodes per world. In both runs, every checkpoint, including step 0, has a 131,072-token budget, 10 experiment rounds, the fitting tool and identical sampling.

Run 1 and run 2 consist of 87 easy world templates, and run 1 has the 96 medium and 88 hard world templates that were available at training time, while run 2 has 148 medium and 152 hard world templates (including the same templates in run 1). We sample from these templates to make unique world instances shown in training. At each training step, we randomly sample 32 instances (except run 2 weights the choice of an easy, medium, or hard world with a split of 45% easy, 45% medium, 10% hard).

The refinement and validation process is simplified to 4 checks:

  1. Runtime validity: the law compiles and runs in the simulator without errors
  2. Robustness: the law is robust to the same initial condition and step size search as in the paper
  3. Reimplementation: the law can be implemented in Python correctly from just the text description
  4. Novelty: an agent checks that the law is different from all other created worlds and DiscoverPhysics worlds

Worlds then still get refined for difficulty. Easy worlds just undergo checks 1 and 2, while medium and hard undergo all 4.

See a figure that shows a schematic of this pipeline, revised from our Fig. 1, that will be included in the paper along with these results: https://postimg.cc/yDXRKTfZ

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.