acceptodds
Under review as a conference paper at ICLR 2027

Scaling Automatic Research Agents via World Models

Abstract

Automating empirical research is a long-standing direction of AI. Recent automatic research (AutoResearch) agents bring this goal within reach, as modern LLMs show the capability to independently implement solutions and learn from the execution outcomes. Behind these gains, post-training (especially RL) plays a central role. In this paper, we identify a fundamental tension when scaling RL for these agents: the two components of every AutoResearch trajectory (agent generation and environment execution) scale in very different manners, since all generation shares compute through batching, while each execution occupies its exclusive sandbox and real machine time. As a result, the environment execution dominates the training cost and becomes the bottleneck as trajectories grow. To resolve this tension, we propose World Model RL (WMRL), which replaces environment execution with a world model to remove this bottleneck. Additionally, the world model can be imperfect, as its rewards are corrupted by bias and noise. Therefore, we further equip WMRL with two mitigations, Online Debiasing and Inverse-Variance Denoising, which offset the bias and suppress the noise respectively. Theoretically, we prove that both mitigations of WMRL improve the convergence guarantee. Empirically, WMRL accelerates training by – on various tasks at different agent scales, while exceeding the performance of standard RL baselines. Moreover, our post-trained 4B and 9B agents outperform much larger open-weight agents of 48B and 120B on held-out benchmarks. Beyond AutoResearch, WMRL also transfers to post-training embodied VLA policies, demonstrating its generalizability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
world models, reinforcement learning, LLM agents, automatic research agents

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_A1E1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WMRL, which uses an LLM-based world model to replace expensive enviroment execution when training AutoResearch agents. The world model predicts code execution results and rewards without actually running the experiemnts. The authors also introduce Online Debiasing and Inverse-Variance Denoising to correct the predicted rewards using a small amount of real feedback. The results show faster training and some performace improvements on AutoResearch and VLA tasks.

Critical strengths

I think the motivation is intresting, since running real experiments is indeed expensive for AutoResearch agents. The proposed idea could be useful for reducing training cost. The paper also provides some theoretical analaysis and relatively complete experiemnts, with a reported 3–4 speedup.

Critical weaknesses
  1. I am quite confused why the proposed method can even outperform RL with real enviroment feedback. The world model is clearly not as reliable as real execution, but WMRL achieves better results than real-environment GRPO. Why? Is it because of some strong prior knowledge, additional regularization, or simply the difference in training? I don't think the current experiemnts provide a convincing explaination for this result. More careful comparisions are needed.

  2. I have serious concerns about the reliability of this so-called world model. The authors basically use an LLM to predict the results of complex ML experiemnts without actually running the code. How can you know the final performance of a model without even training it? This sounds quite unrealistic. Unless the LLM has already seen very similiar experiments during pretraining, I really doubt it can provide reliable predictions. And if it has already seen the results, then what exactly is the model learning here? I think the authors should provide much stronger evidence to support this fundemental assumption.

  3. The generalization ability is also questionable. The world model seems to rely heavily on the pretrainned knowledge of the LLM. It may work on familar datasets or algorithms, but what happens when facing completely new tasks? Also, the proposed bias correction uses a global monotonic mapping, which cannot fix incorrect rankings between different solutions. I don't think the current experiments are enough to prove good generalizaiton. There are too many places where strong priors or overfitting could affect the results.

  4. The VLA experiments are not very convincing either. In this setting, the world model evaluates already executed trajectories instead of actually simluating the robot enviroment. This is quite different from predicting the outcome of an experiment without running it. I don't think such results can demonstate that the proposed method works for complex robotic environments. The improvement is also relatively small.

Questions for authors

Please see the weakness part.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/3)

Dear Reviewer A1E1,

Thank you for the detailed feedback. We answer each weakness below, and we hope they can address your concerns.


[W1] Why WMRL can exceed real-execution GRPO

We appreciate this insightful question. After carefully checking the training trajectories and individual cases, we find that the gap stems from the standard GRPO algorithm: training on real execution systematically makes the agent over-conservative (i.e., it lowers the probability of heavy but effective solutions), as also reported in related work [1][2][3]. We show this through (1) training statistics, (2) the relation between runtime and performance, (3) a simple mitigation, and (4) related literature as follows.

(1) Training statistics. We inspected the MLE-Dojo training trajectories. The agent can attempt heavy strategies (including training a ViT for multiple epochs, training on all raw data without selection, or grid-searching hyperparameters), and heavy strategies account for 84.2% of the timeouts during training. A timed-out turn is treated as a runtime error in MLE-Dojo, so GRPO gives such solutions a negative advantage, while the world model gives a positive one that stays non-negative after Online Debiasing (OD) (Table 1).

Table 1: Avg Advantage (normalized to [-1, 1]) of timed-out heavy solutions throughout training.

GRPOWorld modelWorld model (after OD)
Qwen3.5-4B-0.01+0.55+0.09
Qwen3.5-9B-0.03+0.15+0.02

As a result, GRPO increasingly stops writing heavy solutions as training goes on: the share of trajectories containing a timed-out turn drops by 18% across all tasks and by 41% on image tasks (Table 2).

Table 2: Share of trajectories containing a timed-out turn during Qwen3.5-9B GRPO training.

Training steps1–6667–132133–199
All tasks10.0%8.3%8.2%
Image tasks18.1%13.1%10.6%

(2) Runtime and performance. We then grouped every training execution that started running and finished within the time limit by its runtime (Table 3).

Table 3: Final score of completed training executions, by runtime.

Runtime< 60 s60–120 s120–600 s600–1200 s
Final score17.2%20.9%24.5%25.8%

Longer runs yield better solutions, yet they are also the most likely to hit the time limit, where GRPO truncates them and assigns a lower advantage. As a result, the time limit penalizes exactly the solutions with the highest potential, pushing standard GRPO toward faster but weaker ones.

(3) A simple mitigation. Simply extending the time limit does not solve this problem of GRPO, and it would further increase WMRL's advantage. Any finite limit only moves the threshold, and solutions beyond it are still penalized by standard GRPO. Meanwhile, a longer runtime limit makes each GRPO step wait longer for the rollouts, whereas world model run time doesn't change, so WMRL's speedup would grow further.

(4) Related literature. Such disadvantages of GRPO have already been documented. Yang et al. [1] report that doing GRPO on MLE tasks quickly converges to fast but suboptimal solutions. DAPO [2] also notes that penalizing truncated/timeout samples punishes sound reasoning. Yue et al. [3] find that standard GRPO concentrates the policy on the paths easiest to reward, narrowing the solutions it explores. Our real-execution GRPO for AutoResearch faces the same pressure, whereas WMRL's world model scores a solution by its content rather than its runtime.

We have added this analysis with Tables 1–3 to the appendix and summarized it in Section 5.2.


References

  • [1] Yang et al., Reinforcement Learning for Machine Learning Engineering Agents, ICLR 2026
  • [2] Yu et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale, NeurIPS 2025
  • [3] Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, NeurIPS 2025
AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/3)

[W2] Reliability of the World Model

We agree that the world model is less reliable than real execution, and WMRL does not assume otherwise. Instead, we treat the world model as a noisy but informative reward source and correct it with real execution. We show this through (1) what the world model predicts and whether it matches execution, and (2) how far WMRL's rewards are from the execution rewards.

(1) What the world model predicts. To return the same information as the environment, the world model assigns each solution one of four outcomes, each with its own reward:

  • syntax error (the code itself is invalid): 0;
  • runtime error (the code fails while running): 0.1;
  • no valid submission (the code runs but writes no valid submission): 0.2;
  • valid submission: 0.5 + 0.5 × leaderboard percentile.

Most of these outcomes can be inferred from the code, the structure of the given data files, and the installed libraries without running anything. To illustrate this, our anonymous repository provides three pairs of agent-written solutions, each scored by the world model and by real execution. In Case 1, the world model reads that a solution imports a package missing from the sandbox and predicts the runtime error at the exact line, while it predicts that the corrected solution runs. In Cases 2 and 3, it gives the higher reward to the better of two valid solutions, in the same order as real execution. We further report in Table 4 how well the raw world model output agrees with execution.

Table 4: Per-outcome accuracy of the raw world model against execution. Avg is the mean of the four outcomes (balanced accuracy).

Outcome4B world model9B world model
Syntax error93.5%96.8%
Runtime error70.6%86.0%
No valid submission54.9%56.4%
Valid submission74.7%80.2%
Avg73.4%79.9%

Accuracy is highest on syntax errors (above 93%) and lowest on no valid submission (about 55%), and the 9B world model is more accurate on every outcome. WMRL corrects the remaining mismatches as shown next.

(2) Reward gap before and after correction. We restate Figure 5 of our submission as Table 5 below, which reports the reward bias of the world model before and after Online Debiasing (OD).

Table 5: Reward difference of the world model before and after OD on held-out groups (Qwen3.5-4B).

MLE-DojoDSBench
Raw world model0.2280.227
After OD0.016 (−93%)0.013 (−94%)

Raw world model rewards are biased by about 0.23 on the [0, 1] reward scale, and OD reduces this bias by 93–94%. Overall, the world model is often right but sometimes wrong, so WMRL uses it as a noisy but informative reward source, with OD and Inverse-Variance Denoising (IVD) making this source more reliable.

We have added these cases and Table 4 to the appendix.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 3/3)

[W3] Generalization question, and whether our WMRL can fix

We agree that our designed mapping cannot fix all the ranking errors, but statistically we can still observe that it works on held-out groups, which reduces the reward difference between the world model and execution by ~90% on both MLE-Dojo and DSBench (Table 5). The world model itself also ranks solutions in the right order: within each group, its ranking is positively correlated with real execution (Spearman 0.32–0.42 over all solutions and 0.19–0.32 among valid submissions), and Cases 2 and 3 in our anonymous repository show this on concrete pairs of valid solutions. The test generalization is supported by our experiments in two settings as follows:

(1) Tasks unseen in RL. DSBench is fully disjoint from our post-training tasks. Table 6 restates its average from Table 1 of our submission: WMRL outperforms both the base model and GRPO at both scales.

Table 6: Average score on DSBench (restated from Table 1 of our submission).

MethodQwen3.5-4BQwen3.5-9B
Base model17.1%23.9%
GRPO25.7%31.2%
WMRL28.8%32.8%

(2) Tasks unseen in model pretraining. We post-train on SWE-smith [4] and test on SWE-bench Verified [5]. During training, the used tasks are newly synthesized by SWE-smith's pipeline (thus cannot have been foreseen in pretraining), and the world model judges patches that the agent has just written, so the world model's outcome must be predicted. Table 7 reports the results.

Table 7: Resolved rate (%) on SWE-bench Verified after post-training on SWE-smith.

MethodQwen3.5-4BQwen3.5-9B
Base model21.2%37.0%
GRPO24.5%49.2%
WMRL25.1%49.6%

WMRL still outperforms GRPO at both scales, showing that the world model generalizes to tasks and patches it has never seen in pretraining.


[W4] Scope of the VLA experiment

We agree that our world model does not predict the 'world' next state in VLA experiments; instead, it scores the rollout. We would like to clarify what this experiment is designed to show.

(1) Purpose. Scoring rollouts with a learned model (Simulated signal) is the most common practice in robot learning [6][7][8], and thus we adopt this setting to see if our WMRL can still benefit robot policy models. Typically, Sim (world model) and Real (environment) signals in robotics have complementary strengths: the environment returns an accurate but sparse success bit, while the world model returns a dense but noisy score. The experiment therefore tests whether OD and IVD can combine the two signals and keep the advantages of both, and we did not claim a speedup for VLA in the paper either.

(2) Significance. With 4500 evaluation episodes per method, WMRL improves significantly over both SFT (z = 3.56) and GRPO with the sparse reward (z = 2.67).

We have revised Section 5.4 and the Discussion to state this purpose and scope.

All of these changes and analyses are in the revised PDF, highlighted in blue. We would be glad to discuss any of them further.


References

  • [4] Yang et al., SWE-smith: Scaling Data for Software Engineering Agents, NeurIPS 2025
  • [5] Chowdhury et al., Introducing SWE-bench Verified, OpenAI 2024
  • [6] Ma et al., VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training, ICLR 2023
  • [7] Sontakke et al., RoboCLIP: One Demonstration is Enough to Learn Robot Policies, NeurIPS 2023
  • [8] Rocamonde et al., Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning, ICLR 2024
AuthorsOpenReview ↗

Gentle Reminder: Follow-up on Our Rebuttal

Dear Reviewer A1E1,

Thank you again for your insightful questions, especially on why WMRL can exceed real-execution GRPO. They led us to the timeout analysis, which has clearly strengthened our paper. We would be truly grateful if you could take a look at our responses and let us know whether they have addressed your concerns. For your convenience, we briefly summarize them below:

  1. [W1] Why WMRL exceeds GRPO: Real execution gives a timed-out solution the runtime-error reward, so GRPO learns to avoid heavy but effective solutions. Although longer completed runs score higher (25.8% at 600–1200 s vs. 17.2% below 60 s), trajectories with a timed-out turn drop by 18% across all tasks and by 41% on image tasks during GRPO training, while WMRL does not have this problem. Its world model gives these solutions a non-negative advantage (+0.09/+0.02 at 4B/9B after Online Debiasing (OD), vs. −0.01/−0.03 in GRPO).
  2. [W2] Reliability of the world model: The raw world model predicts execution outcomes with 73.4%/79.9% balanced accuracy (4B/9B), so it is a noisy but informative reward source. OD then reduces its reward bias by 93–94%, from about 0.23 to below 0.02 on the [0, 1] reward scale, which makes it more reliable. To make this concrete, our anonymous repository gives three pairs of solutions, and in all three pairs the world model gives the higher reward to the same solution as real execution.
  3. [W3] Generalization: WMRL outperforms GRPO on DSBench, which is disjoint from training (28.8% vs. 25.7% at 4B, 32.8% vs. 31.2% at 9B), and on SWE-bench Verified after post-training on SWE-smith (25.1% vs. 24.5% and 49.6% vs. 49.2%), showing that the world model generalizes to unseen tasks.
  4. [W4] VLA scope: We restated the scope of the VLA experiment in Section 5.4 to make it clearer. The experiment tests whether OD and Inverse-Variance Denoising (IVD) can combine a sparse environment signal with a dense learned reward. WMRL reaches a 41.2% success rate on LIBERO-Long, against 37.4% for SFT and 38.3% for GRPO, and both gains are significant over 4500 evaluation episodes (z=3.56z = 3.56 and 2.672.67), showing that OD and IVD combine the two signals effectively.

We hope these responses address your concerns, and we would be happy to clarify any point or answer any remaining questions. Thank you very much for your time and thoughtful consideration.

Best,

The Authors

Reviewer_ifaZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies the environment-execution cost as a major bottleneck when scaling reinforcement learning for automatic research agents. It proposes World Model RL (WMRL), which uses a frozen language model to predict the execution outcome of candidate solutions and replaces most expensive real executions with batched world-model inference. To address prediction errors, WMRL uses a small stream of real executions as anchor signals, applying Online Debiasing to correct systematic reward bias and Inverse-Variance Denoising to combine real and predicted gradients. Theoretical analysis shows that world-model bias and noise introduce additional convergence errors, while the proposed corrections reduce these terms. Experiments on MLE-Dojo, DSBench, and LIBERO-Long report 3–4× lower training compute and performance comparable to or better than real-environment RL.

Critical strengths
  1. The paper presents a novel and well-motivated approach to scaling RL for automatic research agents. The distinction between batched agent generation and individually expensive environment execution is important, and the combination of a world model with a small amount of real execution is a promising way to address this asymmetry.

  2. The paper is clearly written and easy to follow. The main narrative is well structured: the authors first identify the execution bottleneck, then introduce world-model rewards, analyze their bias and noise, and finally propose two corresponding correction mechanisms. The figures and notation make the overall method accessible.

  3. The theoretical formulation is clear and closely connected to the proposed method. The decomposition of world-model error into systematic bias and zero-mean noise provides a useful explanation of why pure world-model RL can have a convergence floor, while Online Debiasing and Inverse-Variance Denoising are naturally linked to the two error terms.

Critical weaknesses
  1. The paper does not report how the adaptive real-execution weight behaves during training. Since the authors claim that WMRL can automatically increase the contribution of real execution when the world model becomes unreliable, the paper should show the trajectories of ρ^\hat\rho, the effective real-gradient weight αE\alpha_E, and the anchor residual. Without this information, it is difficult to determine whether the reported improvements mainly come from the world model or from the small set of real anchor groups.

  2. When the real-execution weight becomes dominant, WMRL may still use only a small number of real anchor groups. In that case, the update direction may be closer to the true gradient, but its variance may be much larger than that of standard GRPO, which uses real execution for all groups. Therefore, the claimed fallback to standard GRPO may only hold at the level of gradient direction, while the statistical efficiency and final performance may still deteriorate.

  3. The reported reduction in reward bias does not directly demonstrate that the policy-gradient estimate becomes more accurate. The paper does not report the similarity between world-model gradients and real-environment gradients, or how calibration changes gradient direction and variance. This leaves a gap between the reward-level analysis and the actual optimization behavior.

  4. The theoretical analysis relies on a monotone score-distortion assumption and ideal inverse-variance weighting, whereas the implementation uses a residual-based proxy for the variance ratio. The experiments provide indirect evidence that monotone calibration works, but they do not directly verify ranking preservation or the accuracy of the practical weighting rule.

Questions for authors
  1. Please report training-time curves for ρ^t\hat\rho_t, the effective real-gradient weight αE,t\alpha_{E,t}, and the anchor residual, together with their statistics across random seeds. It would also be useful to report the fraction of training steps for which αE\alpha_E exceeds 50%, 75%, and 90%.

  2. Please clarify, through theoretical discussion or training-log analysis, whether a large αE\alpha_E with a small number of real anchor groups can reduce statistical efficiency or final performance. In particular, does “degradation toward standard GRPO” refer only to the direction of the policy gradient, or does it also imply comparable gradient variance and optimization behavior to full real-environment GRPO?

  3. Please provide cosine similarity, estimation error, or variance comparisons between world-model gradients and real-environment gradients before and after Online Debiasing, so that the effect of reward calibration on the actual policy update can be evaluated.

  4. Please report the rank correlation and ranking-reversal rate between world-model scores and real execution scores, as well as the discrepancy between the residual-based variance proxy and the empirically optimal fusion weight. This would help assess the monotonicity assumption and the practical IVD weighting rule.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/2)

Dear Reviewer ifaZ,

Thank you for the careful and constructive review. Your weaknesses and questions pair up one to one, so we answer them together.


[W1 & Q1] Training behavior of ρ^t\hat\rho_t, αE,t\alpha_{E,t} and the anchor residual

We agree these quantities should be reported. With mEm_E anchor groups and mWMm_{WM} world model groups in a step (mE=1m_E = 1, mWM=7m_{WM} = 7), the effective real-gradient weight is αE=mEρ^/(mEρ^+mWM)\alpha_E = m_E\hat\rho/(m_E\hat\rho + m_{WM}), and the normalized anchor residual is η^t2/η^cal2=ρ^t−1\hat\eta^2_t/\hat\eta^2_{\mathrm{cal}} = \hat\rho_t - 1. Single-step values are noisy, so Tables 1 and 2 average them over five equal windows of the 200 training steps.

Table 1: Window averages of ρ^t\hat\rho_t, the normalized anchor residual and αE,t\alpha_{E,t} (Qwen3.5-9B, MLE-Dojo).

Training Steps0–3940–7980–119120–159160–199
ρ^t\hat\rho_t2.001.541.511.471.29
Normalized residual1.000.540.510.470.29
αE,t\alpha_{E,t}22.2%18.0%17.7%17.4%15.6%

Table 2: Window averages of ρ^t\hat\rho_t, the normalized anchor residual and αE,t\alpha_{E,t} (Qwen3.5-4B, MLE-Dojo).

Training Steps0–3940–7980–119120–159160–199
ρ^t\hat\rho_t2.001.781.721.581.43
Normalized residual1.000.780.720.580.43
αE,t\alpha_{E,t}22.2%20.3%19.7%18.4%17.0%

At both scales, the anchor residual and αE\alpha_E decrease over training, and no step has αE\alpha_E above 50%. As a result, more than 75% of the gradient in WMRL originates from the world model groups throughout training.

We have added these results to the appendix.


[W2 & Q2] Questions on degrading toward GRPO

We answer this question in two parts: whether a large αE\alpha_E affects training and final performance, and what "degrading toward GRPO" means.

(1) Training stability and final performance. We analyzed the training-time metrics to ensure that they remain stable throughout training and that the final performance is not affected.

  • Stable training. As shown in [W1 & Q1], ρ^t\hat\rho_t, the anchor residual and αE\alpha_E remain stable and decrease steadily over training, which shows that the world model continues to play an important role. αE\alpha_E stays between 15.6% and 22.2%, and no step has a sudden spike above 50%.
  • Final performance. Through this stable training and alignment, WMRL effectively combines the signals from the world model and real execution. It therefore outperforms both GRPO with full real execution and RL with a pure world model, by 2.2 and 4.9 points on average on MLE-Dojo and DSBench, and by 2.9 and 2.0 points on LIBERO-Long (Tables 1 and 2 of the paper).

(2) Meaning of "degrading toward GRPO". You are right that it conceptually refers to the environment gradient direction. When αE\alpha_E increases, in the worst case, WMRL still learns from real execution.

We have revised the Discussion to state this precisely and avoid misunderstanding.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/2)

[W3 & Q3] Gradient-level effect of Online Debiasing

We agree that a lower reward bias does not by itself imply a more accurate policy gradient. We therefore measure how Online Debiasing (OD) changes the reliability and the direction of the world model gradient.

(1) Reliability. Since ρ^\hat\rho approximates the variance ratio VWM/VEV_{WM}/V_E, the anchor residual measures how reliable the world model gradient is.

Table 3: Whole-run averages of the normalized anchor residual and ρ^\hat\rho before and after OD (MLE-Dojo).

Residual before ODResidual after ODρ^\hat\rho before ODρ^\hat\rho after OD
Qwen3.5-4B3.470.70 (↓80%)4.471.70 (↓62%)
Qwen3.5-9B2.140.56 (↓74%)3.141.56 (↓50%)

OD thus makes the world model gradient useful for training.

(2) Direction. On the same anchor trajectories, we compute the policy gradient with real rewards (gEg_E), raw world model scores and OD-calibrated scores, averaged over checkpoints every 40 steps.

Table 4: Cosine similarity with the real-environment gradient gEg_E before and after OD (MLE-Dojo).

Raw world modelAfter OD
Qwen3.5-4B0.220.31 (↑41%)
Qwen3.5-9B0.210.29 (↑38%)

OD thus aligns the world model gradient more closely with the real one.

We have added both tables to the appendix.


[W4 & Q4] Verifying monotonicity and the practical weight

We verify the two assumptions through experiments directly: ranking preservation by the world model, and the accuracy of the residual-based weight ρ^\hat\rho.

(1) Ranking preservation. We compute the within-group rank correlations between world model and real scores, and the reversal rate, i.e., the fraction of within-group pairs that the world model orders opposite to real execution.

Table 5: Within-group ranking agreement between world model and real scores (MLE-Dojo).

Spearman (median / mean)Kendall τb\tau_b (median / mean)Reversal rate
Qwen3.5-4B0.37 / 0.320.36 / 0.3215.2%
Qwen3.5-9B0.42 / 0.380.39 / 0.3612.6%

Both correlations are positive and only 15.2%/12.6% of the pairs are reversed, supporting the monotone assumption. Beyond these statistics, Cases 2 and 3 in our anonymous repository show concretely that the world model ranks two valid solutions to the same task in the same order as real execution.

(2) Accuracy of the practical weight. On the same anchor trajectories, we measure the variances of the world model and real gradients to obtain the empirical ratio ρ∗=VWM/VE\rho^\ast = V_{WM}/V_E, and compare it with ρ^\hat\rho. By Remark 8, a relative error ε=∣ρ^−ρ∗∣/ρ∗\varepsilon = \lvert\hat\rho - \rho^\ast\rvert/\rho^\ast in the weight increases the fused variance by at most about ε2/4\varepsilon^2/4.

Table 6: Residual-based weight ρ^\hat\rho versus the empirical variance ratio ρ∗\rho^\ast (MLE-Dojo, whole-run average).

ρ^\hat\rho∣ρ^−ρ∗∣\lvert\hat\rho - \rho^\ast\rvertVariance excess (≤ε2/4\le \varepsilon^2/4)
Qwen3.5-4B1.700.240.68%
Qwen3.5-9B1.560.230.75%

The residual-based weight thus stays close to the empirical optimum and increases the fused variance by less than 1%.

We have added both analyses to the appendix. All changes are highlighted in blue in the revised PDF. We would be glad to discuss further.

AuthorsOpenReview ↗

Gentle Reminder: Follow-up on Our Rebuttal

Dear Reviewer ifaZ,

Thank you again for your detailed review and for recognizing our approach as novel and well motivated. Your questions on the training behavior and on the gradient-level effect of our corrections helped us add new diagnostics that make the paper more complete. We would greatly appreciate it if you could let us know whether our responses have resolved your questions. We summarize the main points below:

  1. [W1 & Q1] Training behavior: To show whether the world model keeps its role during training, we tracked the anchor share αE\alpha_E. It stays between 15.6% and 22.2% and never exceeds 50%, so more than 75% of the gradient comes from the world model and training remains stable.
  2. [W2 & Q2] Degrading toward GRPO: This phrase refers to the environment gradient direction, meaning that even when αE\alpha_E grows, WMRL in the worst case still learns from real execution, and we have revised the Discussion to state this precisely. In practice, αE\alpha_E stays below 25%, and WMRL outperforms both GRPO and RL with a pure world model, by 2.2 and 4.9 points on average on MLE-Dojo and DSBench.
  3. [W3 & Q3] Effect of OD: To show the effect of Online Debiasing (OD) at the gradient level, we measured both the reliability and the direction of the world model gradient. OD reduces the normalized anchor residual by 74–80% (from 3.47/2.14 to 0.70/0.56 at 4B/9B), which makes the gradient more reliable, and raises its cosine similarity with the real-execution gradient by 38–41% (from 0.22/0.21 to 0.31/0.29), so it aligns the world model gradient more closely with the real one.
  4. [W4 & Q4] Assumptions: We verified both assumptions directly. For monotonicity, the within-group Spearman correlation with real scores is positive (mean 0.32/0.38 at 4B/9B), and only 15.2%/12.6% of pairs are ranked in the opposite order, so the world model largely preserves the ranking. For the practical weight, the residual-based weight ρ^\hat\rho increases the fused variance by only 0.68%/0.75% relative to the optimal weight, so it stays close to the empirical optimum.

We hope these results answer your questions. If anything remains unclear, we would be glad to provide further details. Thank you sincerely for your time and valuable feedback.

Best,

The Authors

Reviewer_rYDmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

WMRL uses a frozen language model to predict execution results for most training samples. A small number of real executions are used to adjust rewards and set gradient weights. The authors report training about three times as fast, better results on held-out tasks, and improvements in VLA training.

Critical strengths
  1. Both model sizes train about three times as fast and perform better on held-out tasks.

  2. The ablation results show that both correction components help. The VLA results suggest that reward correction can also help outside research agent training.

Critical weaknesses
  1. The same predicted score always receives the same correction, even when errors differ across tasks. The paper does not explain how calibration samples are grouped or show that a shared monotone correction works across the tasks in each group.

  2. IVD uses reward errors to estimate the gradient variance ratio, but the paper does not test this link. The weights depend on errors relative to their warmup value, so predictions that stay inaccurate may still receive a high weight.

  3. Figure 5 shows lower overall reward bias, but does not show whether the world model can rank valid solutions. The only detailed example is a simple file path error. Its ability to predict solution quality is still unclear.

  4. The paper does not report the total number of rollouts or real environment interactions used by each method. It is therefore unclear whether WMRL's gains over GRPO trained on real feedback come from its feedback design or from differences in sampling budgets.

  5. Lemma 11 needs a bound on E[f(r^)∣τ]\mathbb{E}[f(\hat r)\mid\tau], but the proof seems to bound f(E[r^∣τ])f(\mathbb{E}[\hat r\mid\tau]) instead. For a nonlinear ff, these are generally not equal. This leaves a gap in the claim that calibration reduces bias over time.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/2)

Dear Reviewer rYDm,

Thank you for the comments, in particular of the proof of Lemma 11. Your observation there is correct, and we give the corrected statement under [W5]. For W1–W4, we answer with measurements from our training runs.


[W1] Shared debias function across tasks

A shared map is aligned with our aim, and we show below that it is enough to reduce the reward difference by over 90% on unseen tasks.

In training, all tasks share a single pool of anchor pairs, and the map fitted on it is applied to the world model groups of other tasks, which have no real rewards. To test this transfer (Section 5.6), we split the tasks into two disjoint sets, fit the map on the anchor groups of one set, and compare rewards before and after Online Debiasing (OD) on the groups of the other set:

Table 1: Reward difference of the world model before and after OD on unseen tasks (Qwen3.5-4B).

MLE-DojoDSBench
Raw world model0.2280.227
After OD0.016 (↓93%)0.013 (↓94%)

OD thus remains effective on unseen tasks, reducing the reward difference by 93% on MLE-Dojo and 94% on DSBench.


[W2] The link between reward error and gradient variance in Inverse-Variance Denoising (IVD)

We address the two concerns in turn: whether reward error tracks gradient variance, and whether the warmup normalization can over-weight inaccurate predictions.

(1) Testing the link. On the same anchor trajectories, we measured the variances of the world model and real gradients to obtain the empirical ratio ρ∗=VWM/VE\rho^\ast = V_{WM}/V_E, and compared it with the residual-based ρ^\hat\rho:

Table 2: Residual-based weight ρ^\hat\rho versus the empirical variance ratio ρ∗\rho^\ast (MLE-Dojo, whole-run average).

ρ^\hat\rho∣ρ^−ρ∗∣\lvert\hat\rho - \rho^\ast\rvertVariance excess
Qwen3.5-4B1.700.240.68%
Qwen3.5-9B1.560.230.75%

By Remark 8, the variance excess is ε2/4\varepsilon^2/4 with ε=∣ρ^−ρ∗∣/ρ∗\varepsilon = \lvert\hat\rho-\rho^\ast\rvert/\rho^\ast, so the proxy increases the fused variance by less than 1%.

(2) Warmup normalization. The warmup value only sets the reference scale of ρ^\hat\rho, and ρ^\hat\rho rises whenever the world model becomes less accurate than at warmup. Table 2 confirms that the resulting weight stays close to the measured optimum, so inaccurate predictions are not over-weighted in practice.


[W3] Can the world model rank solutions?

We show that the world model can rank solutions of different quality through (1) its statistical agreement with real execution within each group, and (2) concrete cases.

(1) Statistical agreement. To show that the world model gives different solutions in a group different scores in the correct order, we compare the world model ranking with the real ranking within each group, using rank correlations and the rank reversal rate (the share of solution pairs ranked in the wrong order):

Table 3: Within-group rank agreement between world model and real scores (MLE-Dojo, all tasks).

Rank Spearman, [−1,1][-1, 1] (median / mean)Rank Kendall τb\tau_b, [−1,1][-1, 1] (median / mean)Rank reversal rate
Qwen3.5-4B0.37 / 0.320.36 / 0.3215.2%
Qwen3.5-9B0.42 / 0.380.39 / 0.3612.6%

Table 3b: Within-group rank agreement among valid submissions only (MLE-Dojo, all tasks).

Rank Spearman, [−1,1][-1, 1] (median / mean)Rank Kendall τb\tau_b, [−1,1][-1, 1] (median / mean)Rank reversal rate
Qwen3.5-4B0.24 / 0.190.25 / 0.1921%
Qwen3.5-9B0.31 / 0.270.32 / 0.3019%

Among valid submissions, the correlation is slightly weaker (0.19–0.32, Table 3b), but on a range of −1 to 1 it is still clearly positive; over all solutions, it is stronger (0.32–0.42, Table 3).

(2) Cases. Cases 2 and 3 in our anonymous repository each compare two valid solutions to the same task: one uses a weaker model (Case 2), and one trains a model but never writes its predictions into the submission (Case 3). In both, the world model gives the higher reward to the better solution, matching real execution. We have added these cases to the appendix.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/2)

[W4] Rollout and real-interaction budgets

Both methods train for 200 steps, each with 8 groups of 8 trajectories, and every trajectory takes 4 turns, each executing one solution. Each real execution has a budget of up to 20 minutes (1200 s), so one trajectory can occupy the environment for up to 80 minutes, which is why real execution dominates the training cost. This configuration is the same for 4B and 9B. In WMRL, one of the 8 groups per step is an anchor group:

GRPO (real execution)WMRL
Training steps200200
Real executions512006400 (anchor groups only)
World model grading calls—51200
Training compute, 4B (GPU-hours)883286
Training compute, 9B (GPU-hours)1174349

WMRL therefore uses the same rollouts as GRPO, 8× fewer real executions and 3.1×/3.4× less training compute (Figure 3 of the paper), while outperforming GRPO in Table 1 of the paper. Its gains thus come from the feedback design, not from a larger sampling budget.


[W5] The gap in Lemma 11

Thank you for catching this. The proof bounds f^(E[r^])\hat f(\mathbb{E}[\hat r]), while the lemma concerns E[f^(r^)]\mathbb{E}[\hat f(\hat r)], and the two differ for the piecewise-linear f^\hat f. We have fixed the proof accordingly. Our main theoretical claim, that Online Debiasing makes the bias term of Theorem 4 contractive, still holds, and we show empirically that the error introduced by the gap is negligible, as detailed below.

(1) Theoretically. The difference between E[f^(r^)]\mathbb{E}[\hat f(\hat r)] and f^(E[r^])\hat f(\mathbb{E}[\hat r]) is at most BJ≤LE∣ξ∣B_J \le L\mathbb{E}|\xi|, where LL is the Lipschitz constant of f^\hat f. Adding BJB_J to Lemma 11 only changes the definition of T0T_0 in Theorem 4, from cf2/B2c_f^2/B^2 to (cf2+BJ2T)/B2(c_f^2 + B_J^2 T)/B^2. The bias term keeps its form O~(M2B2/(1+T/T0))\tilde O\big(M^2B^2/(1+T/T_0)\big) and remains contractive.

(2) Empirically. Figure 5 measures the calibrated reward against the true reward, which includes this gap. After OD the difference is 0.016 on MLE-Dojo and 0.013 on DSBench, against 0.228 and 0.227 before, so BJ/B≈0.07B_J/B \approx 0.07 and 0.060.06. The error introduced by the gap is therefore negligible.

We have corrected Lemma 11, its proof, Theorem 4 and Corollary 7 in the revised appendix.

All of these changes and analyses are in the revised PDF, highlighted in blue. We welcome further questions.

AuthorsOpenReview ↗

Gentle Reminder: Follow-up on Our Rebuttal

Dear Reviewer rYDm,

Thank you again for your careful review and your close reading of our theory. Your comments, especially the one on Lemma 11, helped us correct the proof and make our analysis more rigorous. We would be very grateful if you could let us know whether our responses have addressed your concerns. To save your time, here is a short summary:

  1. [W1] Shared debias map: To test whether a single shared map transfers across tasks, we fitted it on one set of tasks and applied Online Debiasing (OD) to a disjoint set. At 4B, the reward difference drops from 0.228 to 0.016 on MLE-Dojo and from 0.227 to 0.013 on DSBench (by 93–94%), so a shared map is enough for unseen tasks.
  2. [W2] IVD weight: To test whether reward error tracks gradient variance in Inverse-Variance Denoising (IVD), we compared the residual-based weight ρ^\hat\rho with the measured variance ratio. They differ by at most 0.24, which increases the fused variance by only 0.68%/0.75% (4B/9B), so inaccurate predictions are not over-weighted in practice.
  3. [W3] Ranking: To test whether the world model can rank solutions, we computed the within-group Spearman correlation between world model and real scores, which ranges from −1 to 1. It is 0.32–0.42 over all solutions and 0.19–0.32 among valid submissions, a clear positive correlation. To make this concrete, Cases 2 and 3 in our anonymous repository each compare two valid solutions, and the world model ranks them in the same order as real execution.
  4. [W4] Budgets: WMRL uses the same rollouts as GRPO but only 6400 real executions instead of 51200 (8× fewer), and 286/349 instead of 883/1174 GPU-hours at 4B/9B (3.1×/3.4× less), so its gains come from the feedback design rather than a larger sampling budget.
  5. [W5] Lemma 11: We fixed the Jensen gap in Lemma 11. The bias term of Theorem 4 remains contractive, so our main theoretical claim still holds. Empirically, the gap is negligible, since the calibrated reward differs from the true reward by only 0.016/0.013 on MLE-Dojo/DSBench, which gives BJ/B≈0.07B_J/B \approx 0.07 and 0.060.06.

We hope this resolves the issues you raised. We would be happy to clarify any remaining point or answer any further questions. Thank you so much for your time and careful consideration.

Best,

The Authors

Reviewer_K4vXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper basically argues that the bottleneck in RL for AutoResearch agents (LLMs that write, run, and iterate on ML solutions) is environment execution, not generation. Generation batches well on vLLM/SGLang, but every candidate solution needs its own GPU sandbox, so execution cost grows linearly with the number of trajectories.

Their fix is World Model RL in some sense: Replace most real executions in GRPO with a world model. The world model is a frozen, prompted copy of the agent’s own base model. It runs 8 static checks on each solution and returns a predicted score plus simulated env feedback for the next turn. 10% of groups ( a la anchor groups) still get graded by real execution. Those paired scores are used to correct the world model’s reward error, which they model as bias + zero-mean noise.

They add two more corrections on top of this. Online Debiasing: an isotonic map from predicted -> real scores, refit during training and Inverse-Variance Denoising: fuses the anchor and world-model gradients with inverse-variance weights, which in practice are estimated from the calibration residuals.

Critical strengths

The framing is pretty clean: generation amortizes through batching and execution doesn’t, so for MLE/data-science agents the sandbox cost is what caps RL scale. I think this is a real and growing problem for anyone doing RL on execution-heavy agentic tasks.

They measure the fix end-to-end in GPU-hours against the baseline it’s meant to replace (real-execution GRPO). All runs use identical A100 allocations and share the scaffold, hyperparameters and data split. Compute goes 883 -> 286 GPU-hours (4B) and 1174 -> 349 (9B), and held-out scores are higher on all 4 benchmark averages, not lower.

Basically, this is the strongest point in the paper’s favor for me. A ~3x cheaper RL loop with no performance tax on held-out tasks would change what’s affordable here. That includes DSBench, which is fully disjoint from training. This assumes the compute accounting holds up (pls see weaknesses).

Critical weaknesses

The 3.1x / 3.4x is just total GPU-hours (883 -> 286, 1174 -> 349) under “identical A100 allocations”. The paper doesn’t say: how many GPUs the allocation has, or how they’re split between inference and sandboxes, how many training steps each run took, how the cost breaks down into generation, world model inference, execution and training.

The anchor numbers make me a bit worried here: Pure world model RL is ~269 / ~330 GPU-hours, since WMRL is only 6.3% / 5.8% more. So the anchor groups add ~17 / ~19 GPU-hours. Anchors are ~10% of groups (at least 1 of 8 per step per Appendix D). At that rate, executing every group would cost roughly 140-190 GPU-hours. But GRPO is 614 / 844 GPU-hours more than pure world model. That gap is a lower bound on GRPO’s execution cost, since the pure world model run also pays for world model inference. So if both ran the same number of steps, real execution in WMRL is somehow at least ~3.5x cheaper per execution than in GRPO.

I can think of reasons for this: anchor execution overlapped with world model inference, GRPO queuing on a fixed allocation like in Figure 1c, or different step counts. But none of it is described, and each one would make the 3x depend on scheduling and allocation choices rather than on replacing execution itself. The VLA experiment also reports no compute, so the speedup is only shown for training on MLE-Dojo tasks.

Basically, the speedup is the main contribution and the main thing in the paper’s favor for me, so if I can’t tell where it comes from, I can’t really evaluate the main claim. I’d want the per-component breakdown, step counts and allocation size for GRPO, pure world model and WMRL.

The claim that WMRL beats real execution GRPO isn’t statistically supported, and the paper doesn’t explain why it would happen.

The gains over GRPO are +1.2 / +2.8 on MLE-Dojo and +3.1 / +1.6 on DSBench, with no seeds or CIs reported. As far as I can tell, it’s one training run per config. The MLE-Dojo test set is only 14 tasks, so each category average is over 4-5 tasks, and some categories already go the other way (4B MLE-Dojo Text -0.5, 9B DSBench Regression -1.2).

This matters because “matches or exceeds real execution RL” is the other half of the main claim. If it’s really “matches within noise”, the paper is still useful, but then it basically comes down to the speedup (pls see above). If WMRL really beats GRPO with ~8-10x fewer real rewards, that needs an explanation. I’d want at least 3 seeds with CIs for GRPO and WMRL, and held-out score vs step curves at matched steps.

Questions for authors

How many training steps did GRPO, pure world model and WMRL each run, and were they matched on steps, wall-clock or something else? Could you also share: the GPU allocation size, how it’s split between inference and the execution sandboxes, a GPU-hour breakdown into generation, world model inference, execution and training for each method.

How are the anchor groups actually run? Specifically: Are anchor executions synchronous inside each step, or overlapped with world model inference or the next step? The anchors only add ~17 / ~19 GPU-hours over pure world model while GRPO is 614 / 844 more, so I think some overlap is happening. If anchor gradients end up coming from an older policy, how is that handled in the fusion?

Does an anchor group interact with the real environment at every turn, or only execute the final solution?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/2)

Dear Reviewer K4vX,

Thank you for the positive and constructive feedback, and for recognizing both the execution bottleneck and our end-to-end compute measurement. As you point out, the speedup is our main claim, so we first give the full compute accounting and then answer the remaining questions.


[W1 & Q1] Allocation, step counts and time breakdown

Allocation. GRPO and WMRL use the same 16 A100 GPUs on two servers and both train for 200 steps:

Server 1 (8 GPUs)Server 2 (8 GPUs)
GRPO4 for the trainer, 4 for rollout8 for 16 environments in parallel
WMRL4 for the trainer, 4 for rollout6 for 12 environments in parallel, 2 for the world model

Pure world model RL runs on the same two servers for 200 steps, without real execution.

Time breakdown. Training runs as a fully asynchronous pipeline: while the trainer updates the policy on the trajectories of one step, the rollout side already collects the trajectories of the next step with the previous policy (one-step off-policy). Trajectory collection and policy training therefore overlap, and the time of each step is

step time=max⁡(trajectory collection,policy training)+launch and sync overhead.\text{step time} = \max(\text{trajectory collection}, \text{policy training}) + \text{launch and sync overhead}.

Table 1 reports each term, averaged over the 200 steps.

Table 1: Average time per training step over 200 steps. Step time = max(collection, training) + overhead.

Trajectory collection (generation + scoring)Policy trainingLaunch & sync overheadOverall Step time
GRPO, 4B990 s148 s~3 s993 s
GRPO, 9B1315 s357 s~5 s1321 s
WMRL, 4B319 s146 s~3 s322 s
WMRL, 9B374 s390 s~3 s393 s

For example, GRPO at 4B takes max(990 s, 148 s) + 3 s = 993 s per step. Multiplying the step time by 200 steps and 16 GPUs reproduces the GPU-hours in the paper: 883 → 286 (4B) and 1174 → 349 (9B).

Trajectory collection runs in parallel across trajectories, while generation and scoring alternate serially within each trajectory, and each trajectory also pays cold-start costs for generation and the environment. We therefore report collection as a whole rather than split it further. Replacing most real executions cuts collection time by 3.1× (4B) and 3.5× (9B); at 9B, WMRL becomes bound by policy training.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/2)

[W2 & Q2] Why anchors look cheap, and stale anchors

Anchor fraction. Each step has 8 groups, and exactly one of them is an anchor group (12.5%).

Why anchors add little time. Execution cost does not grow linearly with the number of executions, because executions queue on a fixed number of environments. In GRPO, even with fully asynchronous scheduling, where a free environment immediately takes the next execution, each turn's 64 trajectories still queue for 16 environments, and a single execution can take up to 20 minutes. In WMRL, the 8 anchor trajectories fit into the 12 environments at once and run concurrently with world model scoring of the other 7 groups. As a result, WMRL takes only about 20 s per step more than pure world model RL (322 vs. 303 s at 4B, 393 vs. 371 s at 9B), matching the 6.3%/5.8% extra compute in Section 5.3. This queuing under a fixed allocation is exactly the bottleneck WMRL targets (Figure 1c).

Stale anchors. With one-step off-policy updates, anchor groups lag the current policy by at most one version, as all other groups do. This gap is handled by the PPO-style importance ratio and clipping, and the fusion weights do not depend on it.


[Q3] Do anchor groups interact with the real environment at every turn?

Yes. Once a group is assigned to real execution, all 4 turns of all 8 trajectories in it are executed in the environment and receive real feedback. The world model also scores each anchor turn, which yields the calibration pairs.


[W3] Compute for the VLA experiment

We do not claim a speedup for VLA. In this experiment, every rollout is still executed in simulation, and the world model scores it as a learned reward. The experiment instead tests whether Online Debiasing (OD) and Inverse-Variance Denoising (IVD) can combine the sparse environment signal with the dense world model signal and keep the advantages of both, as stated in the revised Section 5.4.


[W4] Statistical support, and why WMRL can exceed GRPO

We answer this in two parts: (1) statistical support across seeds, and (2) why WMRL can exceed real-execution GRPO.

(1) Statistical support. We ran 3 seeds for GRPO and WMRL at both scales, and report the difference with its 95% confidence interval (Welch's t-test):

Table 2: GRPO and WMRL over 3 seeds (mean ± std), and the 95% CI of their difference.

GRPOWMRLWMRL − GRPO [95% CI]
MLE-Dojo, 4B15.2 ± 0.5416.4 ± 0.67+1.2 [−0.2, +2.6]
MLE-Dojo, 9B18.8 ± 0.3721.6 ± 0.70+2.8 [+1.4, +4.2]
DSBench, 4B25.7 ± 0.5028.8 ± 0.41+3.1 [+2.1, +4.2]
DSBench, 9B31.2 ± 0.5232.8 ± 0.49+1.6 [+0.5, +2.8]

WMRL improves on GRPO in all four settings, by clearly more than the seed-to-seed standard deviation (0.37–0.70). On LIBERO-Long, WMRL also improves significantly over GRPO (z = 2.67 over 4500 evaluation episodes).

(2) Why WMRL can exceed GRPO. Training on real execution makes GRPO over-conservative. A timed-out turn receives the runtime-error reward, so GRPO gives heavy but effective solutions (e.g., training a ViT for multiple epochs) a negative advantage, while the world model scores a solution by its content rather than its runtime. In 9B GRPO training, the share of trajectories with a timed-out turn drops from 10.0% to 8.2% overall and from 18.1% to 10.6% on image tasks, although longer completed executions score higher (17.2% below 60 s vs. 25.8% at 600–1200 s). This matches prior reports that RL on MLE tasks drifts toward fast but suboptimal solutions (Yang et al., ICLR 2026) and that penalizing truncated samples punishes sound reasoning (DAPO, NeurIPS 2025).

We have added the seed results and this analysis to the revised paper.

Thank you again for this review, which made the accounting much clearer. All changes are highlighted in blue in the revised PDF.

AuthorsOpenReview ↗

Gentle Reminder: Follow-up on Our Rebuttal

Dear Reviewer K4vX,

Thank you again for your constructive and positive review. Your questions on the compute accounting helped us make the efficiency comparison with GRPO much clearer and more transparent. We would really appreciate it if you could let us know whether our responses have resolved your concerns. Here is a brief summary:

  1. [W1 & Q1] Compute accounting: Both methods use the same 16 A100 GPUs and 200 steps in a fully asynchronous pipeline, where step time = max(trajectory collection, policy training) + launch and sync overhead. The measured step times are 993 s vs. 322 s at 4B and 1321 s vs. 393 s at 9B, and multiplying them by 200 steps and 16 GPUs reproduces the GPU-hours in the paper (883 → 286 and 1174 → 349), so the 3.1×/3.4× reduction follows directly from the measured step times.
  2. [W2 & Q2, Q3] Anchors: Anchor groups execute every turn in the real environment, but they run concurrently with world model scoring. WMRL therefore takes only about 20 s per step more than pure world model RL (322 vs. 303 s at 4B, 393 vs. 371 s at 9B), so anchors add little cost.
  3. [W3] VLA compute: We restated the scope of the VLA experiment in Section 5.4 to make it clearer. Every rollout is still executed in simulation, so the experiment does not measure training speed. Instead, it tests whether Online Debiasing (OD) and Inverse-Variance Denoising (IVD) can combine the sparse environment signal with the dense world model signal, and WMRL reaches a 41.2% success rate on LIBERO-Long, against 37.4% for SFT and 38.3% for GRPO.
  4. [W4] Statistical support: To test whether the gains are robust, we repeated training over three seeds. WMRL improves on GRPO by 1.2–3.1 points in all four settings, more than the seed-to-seed standard deviation (0.37–0.70), with 95% confidence intervals of the improvement such as +2.8 [+1.4, +4.2] on MLE-Dojo 9B and +3.1 [+2.1, +4.2] on DSBench 4B (all four settings are in Table 2 of our earlier response). On LIBERO-Long, the gain over GRPO is also significant over 4500 evaluation episodes (z=2.67z = 2.67, p<0.01p < 0.01). WMRL can exceed GRPO because real execution gives timed-out solutions the runtime-error reward: in 9B GRPO training, the share of trajectories with a timed-out turn drops from 10.0% to 8.2% overall and from 18.1% to 10.6% on image tasks, even though longer completed runs score higher (25.8% vs. 17.2%).

We hope this addresses the points you raised, and we would be glad to clarify anything further. We truly appreciate your time and your support of our work.

Best,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Rebuttal Summary

We thank the Area Chair and all reviewers for their time and thoughtful feedback. The reviews have helped us clarify the compute accounting, add analyses of the world model and of both corrections, and fix the gap in Lemma 11.

Below, we summarize the key contributions of our paper and the core improvements during the rebuttal:


Contributions

  • We propose WMRL (Reviewers A1E1, ifaZ, K4vX), an RL framework for AutoResearch agents that replaces most environment executions with a world model, while a small anchor stream of real execution keeps it grounded.
  • Online Debiasing (OD) and Inverse-Variance Denoising (IVD) (Reviewers ifaZ, rYDm), which correct the bias and the noise of the world model reward, respectively, each improving one term of the convergence bound.
  • Strong experimental performance (Reviewers A1E1, rYDm, K4vX), with 3.1×/3.4× less training compute than real-execution GRPO at 4B/9B, higher scores on MLE-Dojo and DSBench, and gains in VLA post-training.

Rebuttal Summary

During the rebuttal, we addressed these major concerns:

  • Why WMRL Exceeds GRPO · Reliability of World Models (Reviewer A1E1): We showed that real execution gives a timed-out solution the runtime-error reward, so GRPO learns to avoid heavy but effective solutions: during GRPO training, the share of trajectories with a timed-out turn drops by 18% across all tasks and by 41% on image tasks, while WMRL does not have this problem, as its world model scores a solution by its content rather than its runtime. We also reported the world model's outcome accuracy (73.4%/79.9% accuracy at 4B/9B) and added three pairs of example solutions in our anonymous repository. WMRL also outperforms GRPO on DSBench, which is disjoint from training, and on SWE-bench Verified after post-training on SWE-smith.

  • Training Behavior of De-noise · Gradient-Level Effect of De-bias (Reviewer ifaZ): We showed that αE\alpha_E stays between 15.6% and 22.2%, so more than 75% of the gradient comes from the world model, and clarified that "degrading toward GRPO" refers to the environment gradient direction. OD reduces the anchor residual by 74–80% and raises the cosine similarity with the real-execution gradient by 38–41%.

  • Effectiveness of De-bias · Lemma 11 (Reviewer rYDm): We showed that a single shared map reduces the reward difference by 93–94% on unseen tasks, and that the within-group ranking of the world model is positively correlated with real execution (Spearman 0.32–0.42 over all solutions, 0.19–0.32 among valid submissions). WMRL uses the same rollouts as GRPO and 8× fewer real executions. We fixed the Jensen gap in Lemma 11; the bias term of Theorem 4 remains contractive.

  • Compute Accounting · Statistical Support (Reviewer K4vX): We detailed the GPU layout and the fully asynchronous pipeline, where step time = max(trajectory collection, policy training) + launch&sync overhead. We provided detailed per-step times that reproduce the GPU-hours in the paper, and WMRL's anchors add only about 20 s per step on average. Over three seeds, WMRL improves on GRPO in all four settings, by more than the seed-to-seed standard deviation. We also clarified that we do not claim a speedup for VLA.


Revision Summary

All changes are highlighted in blue in the revised PDF.

  1. Main text: clarified the meaning of "degrading toward GRPO" and the VLA scope (Sections 3.4, 5.4), added the seed results and the timeout explanation (Section 5.2), and pointed to the new gradient diagnostics (Section 5.6).
  2. Appendix B: corrected Lemma 11, the time scale T0T_0 in Theorem 4, and Corollary 7.
  3. New Appendix E: timeout analysis, seed statistics, world model accuracy and ranking, training behavior of IVD, gradient-level effect of OD, compute accounting, and SWE-bench Verified results.

We hope this summary shows that the paper makes both theoretical and practical contributions and that our responses address the reviewers' concerns. We sincerely appreciate the reviewers' and the AC's time and effort, and we would be glad to discuss any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.