- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes WMRL, which uses an LLM-based world model to replace expensive enviroment execution when training AutoResearch agents. The world model predicts code execution results and rewards without actually running the experiemnts. The authors also introduce Online Debiasing and Inverse-Variance Denoising to correct the predicted rewards using a small amount of real feedback. The results show faster training and some performace improvements on AutoResearch and VLA tasks.
Critical strengths
I think the motivation is intresting, since running real experiments is indeed expensive for AutoResearch agents. The proposed idea could be useful for reducing training cost. The paper also provides some theoretical analaysis and relatively complete experiemnts, with a reported 3–4 speedup.
Critical weaknesses
-
I am quite confused why the proposed method can even outperform RL with real enviroment feedback. The world model is clearly not as reliable as real execution, but WMRL achieves better results than real-environment GRPO. Why? Is it because of some strong prior knowledge, additional regularization, or simply the difference in training? I don't think the current experiemnts provide a convincing explaination for this result. More careful comparisions are needed.
-
I have serious concerns about the reliability of this so-called world model. The authors basically use an LLM to predict the results of complex ML experiemnts without actually running the code. How can you know the final performance of a model without even training it? This sounds quite unrealistic. Unless the LLM has already seen very similiar experiments during pretraining, I really doubt it can provide reliable predictions. And if it has already seen the results, then what exactly is the model learning here? I think the authors should provide much stronger evidence to support this fundemental assumption.
-
The generalization ability is also questionable. The world model seems to rely heavily on the pretrainned knowledge of the LLM. It may work on familar datasets or algorithms, but what happens when facing completely new tasks? Also, the proposed bias correction uses a global monotonic mapping, which cannot fix incorrect rankings between different solutions. I don't think the current experiments are enough to prove good generalizaiton. There are too many places where strong priors or overfitting could affect the results.
-
The VLA experiments are not very convincing either. In this setting, the world model evaluates already executed trajectories instead of actually simluating the robot enviroment. This is quite different from predicting the outcome of an experiment without running it. I don't think such results can demonstate that the proposed method works for complex robotic environments. The improvement is also relatively small.
Questions for authors
Please see the weakness part.
4 comments
Rebuttal by Authors (Part 2/3)
[W2] Reliability of the World Model
We agree that the world model is less reliable than real execution, and WMRL does not assume otherwise. Instead, we treat the world model as a noisy but informative reward source and correct it with real execution. We show this through (1) what the world model predicts and whether it matches execution, and (2) how far WMRL's rewards are from the execution rewards.
(1) What the world model predicts. To return the same information as the environment, the world model assigns each solution one of four outcomes, each with its own reward:
- syntax error (the code itself is invalid): 0;
- runtime error (the code fails while running): 0.1;
- no valid submission (the code runs but writes no valid submission): 0.2;
- valid submission: 0.5 + 0.5 × leaderboard percentile.
Most of these outcomes can be inferred from the code, the structure of the given data files, and the installed libraries without running anything. To illustrate this, our anonymous repository provides three pairs of agent-written solutions, each scored by the world model and by real execution. In Case 1, the world model reads that a solution imports a package missing from the sandbox and predicts the runtime error at the exact line, while it predicts that the corrected solution runs. In Cases 2 and 3, it gives the higher reward to the better of two valid solutions, in the same order as real execution. We further report in Table 4 how well the raw world model output agrees with execution.
Table 4: Per-outcome accuracy of the raw world model against execution. Avg is the mean of the four outcomes (balanced accuracy).
| Outcome | 4B world model | 9B world model |
|---|---|---|
| Syntax error | 93.5% | 96.8% |
| Runtime error | 70.6% | 86.0% |
| No valid submission | 54.9% | 56.4% |
| Valid submission | 74.7% | 80.2% |
| Avg | 73.4% | 79.9% |
Accuracy is highest on syntax errors (above 93%) and lowest on no valid submission (about 55%), and the 9B world model is more accurate on every outcome. WMRL corrects the remaining mismatches as shown next.
(2) Reward gap before and after correction. We restate Figure 5 of our submission as Table 5 below, which reports the reward bias of the world model before and after Online Debiasing (OD).
Table 5: Reward difference of the world model before and after OD on held-out groups (Qwen3.5-4B).
| MLE-Dojo | DSBench | |
|---|---|---|
| Raw world model | 0.228 | 0.227 |
| After OD | 0.016 (−93%) | 0.013 (−94%) |
Raw world model rewards are biased by about 0.23 on the [0, 1] reward scale, and OD reduces this bias by 93–94%. Overall, the world model is often right but sometimes wrong, so WMRL uses it as a noisy but informative reward source, with OD and Inverse-Variance Denoising (IVD) making this source more reliable.
We have added these cases and Table 4 to the appendix.
Rebuttal by Authors (Part 3/3)
[W3] Generalization question, and whether our WMRL can fix
We agree that our designed mapping cannot fix all the ranking errors, but statistically we can still observe that it works on held-out groups, which reduces the reward difference between the world model and execution by ~90% on both MLE-Dojo and DSBench (Table 5). The world model itself also ranks solutions in the right order: within each group, its ranking is positively correlated with real execution (Spearman 0.32–0.42 over all solutions and 0.19–0.32 among valid submissions), and Cases 2 and 3 in our anonymous repository show this on concrete pairs of valid solutions. The test generalization is supported by our experiments in two settings as follows:
(1) Tasks unseen in RL. DSBench is fully disjoint from our post-training tasks. Table 6 restates its average from Table 1 of our submission: WMRL outperforms both the base model and GRPO at both scales.
Table 6: Average score on DSBench (restated from Table 1 of our submission).
| Method | Qwen3.5-4B | Qwen3.5-9B |
|---|---|---|
| Base model | 17.1% | 23.9% |
| GRPO | 25.7% | 31.2% |
| WMRL | 28.8% | 32.8% |
(2) Tasks unseen in model pretraining. We post-train on SWE-smith [4] and test on SWE-bench Verified [5]. During training, the used tasks are newly synthesized by SWE-smith's pipeline (thus cannot have been foreseen in pretraining), and the world model judges patches that the agent has just written, so the world model's outcome must be predicted. Table 7 reports the results.
Table 7: Resolved rate (%) on SWE-bench Verified after post-training on SWE-smith.
| Method | Qwen3.5-4B | Qwen3.5-9B |
|---|---|---|
| Base model | 21.2% | 37.0% |
| GRPO | 24.5% | 49.2% |
| WMRL | 25.1% | 49.6% |
WMRL still outperforms GRPO at both scales, showing that the world model generalizes to tasks and patches it has never seen in pretraining.
[W4] Scope of the VLA experiment
We agree that our world model does not predict the 'world' next state in VLA experiments; instead, it scores the rollout. We would like to clarify what this experiment is designed to show.
(1) Purpose. Scoring rollouts with a learned model (Simulated signal) is the most common practice in robot learning [6][7][8], and thus we adopt this setting to see if our WMRL can still benefit robot policy models. Typically, Sim (world model) and Real (environment) signals in robotics have complementary strengths: the environment returns an accurate but sparse success bit, while the world model returns a dense but noisy score. The experiment therefore tests whether OD and IVD can combine the two signals and keep the advantages of both, and we did not claim a speedup for VLA in the paper either.
(2) Significance. With 4500 evaluation episodes per method, WMRL improves significantly over both SFT (z = 3.56) and GRPO with the sparse reward (z = 2.67).
We have revised Section 5.4 and the Discussion to state this purpose and scope.
All of these changes and analyses are in the revised PDF, highlighted in blue. We would be glad to discuss any of them further.
References
- [4] Yang et al., SWE-smith: Scaling Data for Software Engineering Agents, NeurIPS 2025
- [5] Chowdhury et al., Introducing SWE-bench Verified, OpenAI 2024
- [6] Ma et al., VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training, ICLR 2023
- [7] Sontakke et al., RoboCLIP: One Demonstration is Enough to Learn Robot Policies, NeurIPS 2023
- [8] Rocamonde et al., Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning, ICLR 2024
Gentle Reminder: Follow-up on Our Rebuttal
Dear Reviewer A1E1,
Thank you again for your insightful questions, especially on why WMRL can exceed real-execution GRPO. They led us to the timeout analysis, which has clearly strengthened our paper. We would be truly grateful if you could take a look at our responses and let us know whether they have addressed your concerns. For your convenience, we briefly summarize them below:
- [W1] Why WMRL exceeds GRPO: Real execution gives a timed-out solution the runtime-error reward, so GRPO learns to avoid heavy but effective solutions. Although longer completed runs score higher (25.8% at 600–1200 s vs. 17.2% below 60 s), trajectories with a timed-out turn drop by 18% across all tasks and by 41% on image tasks during GRPO training, while WMRL does not have this problem. Its world model gives these solutions a non-negative advantage (+0.09/+0.02 at 4B/9B after Online Debiasing (OD), vs. −0.01/−0.03 in GRPO).
- [W2] Reliability of the world model: The raw world model predicts execution outcomes with 73.4%/79.9% balanced accuracy (4B/9B), so it is a noisy but informative reward source. OD then reduces its reward bias by 93–94%, from about 0.23 to below 0.02 on the [0, 1] reward scale, which makes it more reliable. To make this concrete, our anonymous repository gives three pairs of solutions, and in all three pairs the world model gives the higher reward to the same solution as real execution.
- [W3] Generalization: WMRL outperforms GRPO on DSBench, which is disjoint from training (28.8% vs. 25.7% at 4B, 32.8% vs. 31.2% at 9B), and on SWE-bench Verified after post-training on SWE-smith (25.1% vs. 24.5% and 49.6% vs. 49.2%), showing that the world model generalizes to unseen tasks.
- [W4] VLA scope: We restated the scope of the VLA experiment in Section 5.4 to make it clearer. The experiment tests whether OD and Inverse-Variance Denoising (IVD) can combine a sparse environment signal with a dense learned reward. WMRL reaches a 41.2% success rate on LIBERO-Long, against 37.4% for SFT and 38.3% for GRPO, and both gains are significant over 4500 evaluation episodes ( and ), showing that OD and IVD combine the two signals effectively.
We hope these responses address your concerns, and we would be happy to clarify any point or answer any remaining questions. Thank you very much for your time and thoughtful consideration.
Best,
The Authors
Rebuttal by Authors (Part 1/3)
Dear Reviewer A1E1,
Thank you for the detailed feedback. We answer each weakness below, and we hope they can address your concerns.
[W1] Why WMRL can exceed real-execution GRPO
We appreciate this insightful question. After carefully checking the training trajectories and individual cases, we find that the gap stems from the standard GRPO algorithm: training on real execution systematically makes the agent over-conservative (i.e., it lowers the probability of heavy but effective solutions), as also reported in related work [1][2][3]. We show this through (1) training statistics, (2) the relation between runtime and performance, (3) a simple mitigation, and (4) related literature as follows.
(1) Training statistics. We inspected the MLE-Dojo training trajectories. The agent can attempt heavy strategies (including training a ViT for multiple epochs, training on all raw data without selection, or grid-searching hyperparameters), and heavy strategies account for 84.2% of the timeouts during training. A timed-out turn is treated as a runtime error in MLE-Dojo, so GRPO gives such solutions a negative advantage, while the world model gives a positive one that stays non-negative after Online Debiasing (OD) (Table 1).
Table 1: Avg Advantage (normalized to [-1, 1]) of timed-out heavy solutions throughout training.
As a result, GRPO increasingly stops writing heavy solutions as training goes on: the share of trajectories containing a timed-out turn drops by 18% across all tasks and by 41% on image tasks (Table 2).
Table 2: Share of trajectories containing a timed-out turn during Qwen3.5-9B GRPO training.
(2) Runtime and performance. We then grouped every training execution that started running and finished within the time limit by its runtime (Table 3).
Table 3: Final score of completed training executions, by runtime.
Longer runs yield better solutions, yet they are also the most likely to hit the time limit, where GRPO truncates them and assigns a lower advantage. As a result, the time limit penalizes exactly the solutions with the highest potential, pushing standard GRPO toward faster but weaker ones.
(3) A simple mitigation. Simply extending the time limit does not solve this problem of GRPO, and it would further increase WMRL's advantage. Any finite limit only moves the threshold, and solutions beyond it are still penalized by standard GRPO. Meanwhile, a longer runtime limit makes each GRPO step wait longer for the rollouts, whereas world model run time doesn't change, so WMRL's speedup would grow further.
(4) Related literature. Such disadvantages of GRPO have already been documented. Yang et al. [1] report that doing GRPO on MLE tasks quickly converges to fast but suboptimal solutions. DAPO [2] also notes that penalizing truncated/timeout samples punishes sound reasoning. Yue et al. [3] find that standard GRPO concentrates the policy on the paths easiest to reward, narrowing the solutions it explores. Our real-execution GRPO for AutoResearch faces the same pressure, whereas WMRL's world model scores a solution by its content rather than its runtime.
We have added this analysis with Tables 1–3 to the appendix and summarized it in Section 5.2.
References