acceptodds
Under review as a conference paper at ICLR 2027

Reward-rate Policy Gradient for Efficient Machine Learning Engineering Agents

Abstract

Traditional reinforcement learning (RL) techniques focus on maximizing expected cumulative reward, where each action assumes to take a constant unit of time. However, this assumption does not hold for agentic RL tasks such as machine learning engineering (MLE) agents, where actions involve data loading, feature engineering, and model training that take variable durations. Efficiency matters in modern agentic RL where actions are costly. To address this limitation, we adapt from continuous-time RL and semi-Markov decision process (SMDP) formulation and propose Reward-rate Policy Gradient (RPG), where we focus on optimizing the reward rate — the long-term reward per unit of time. RPG estimates the reward rate from off-policy samples, then charges each action for the time it consumes at that rate. We first conduct theoretical analysis in the bandit setting to establish that RPG approximates the optimal reward rate and empirically demonstrate it outperforms baselines while avoiding enumeration over the policy space, a known issue for an existing method. We then further apply RPG on a small language model (Qwen3.5-4B) with self-improvement loops and empirically show it obtains higher rewards within a fixed time budget than vanilla RL on MLE-bench and NanoGPT, with a 19.2% and 85.7% margin, respectively. Our method provides a practical solution for optimizing performance under wait time considerations in modern agentic RL tasks, where actions interact with external environments and cost time.

Primary area:
reinforcement learning
Keywords:
Machine learning engineering, language model agents, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MwepOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates reward-rate optimization for MLE agents in RL, where actions have different execution times. The proposed RPG estimates the policy's reward rate from historical samples and uses it to penalize actions according to their duration. The paper provide convergence results in a contextual bandit setting. The method is then evaluated on MLE-Bench Lite tasks and NanoGPT and report improvements over vanilla RL under fixed execution-time budgets.

Critical strengths
  1. The paper studies a relevant problem for MLE agents, where action execution time can vary.
  2. The empirical results are extensive and the method is simple enough to be included in standard RL training procedures
Critical weaknesses
  1. The theoretical results are established in a setting and with assumptions that are not established for the proposed algorithm. More precisely, Thm. 1 assumes controlled approximation error for the greedy reward rate while Thm. 2 assumes exact rate estimation and expected action values. The algorithm instead uses posterior-based estimator and PPO.
  2. The proposed experimental results do not isolate the effect of the proposed policy gradient objective.
Questions for authors
  1. Could you isolate the effects of the modified PPO reward and that of the buffer-selection?
  2. How does RPG compare with RL using the same time-aware buffer selection and a fixed coefficient for the time penalty?
  3. Thm. 1 assumes a bounded approximation error for the greedy reward rate. Since the optimlal poilcies are known in the synthetic bandit experiments, can you report the estimation error of the NIW and show that the condition in Thm. 1 is satisfied?
5 comments

We thank the reviewer for the constructive feedback. Please see our responses to each of the raised concerns below.

Weakness 1

Weakness 1: More precisely, Thm. 1 assumes controlled approximation error for the greedy reward rate while Thm. 2 assumes exact rate estimation and expected action values. The algorithm instead uses posterior-based estimator and PPO.

We would like to kindly highlight that our motivation for the theoretical analysis is to study the training dynamics of reward-rate RL by separating the problem of finite-sample estimation from training. In particular, we believe having an accurate finite-sample reward-rate estimate is a problem of estimating a ratio of expectation of sum of random variables over time, which is not central to studying what is unique about the reward-rate RL training dynamics.

For this reason, our analysis is focused on answering these questions under exact expected values: (Q1) under what conditions can we obtain an approximate (or exact) value of optimal ρ∗\rho* through iterating the policy, since ρ∗\rho* is central to finding the problem's solution; (Q2) even with exact estimation, what is unique about reward-rate RL's training.

To that end, Thm.1 answers (Q1) by showing that the iteration on the greedy reward rate converges near ρ∗\rho* when γ≠0\gamma \neq 0 and to exactly ρ∗\rho* when γ=0\gamma = 0. To answer (Q2), Thm.2 further analyzes the training dynamics when that estimation error is removed (γ=0\gamma = 0), which informs us that reward-rate RL has its unique coupling issue even with exact estimation, and that issue is addressed through natural policy gradient.

The posterior-based estimator and PPO is an extension of our theoretical finding into practice. The posterior estimate approximates the greedy reward-rate by fitting on greedy samples ranked by r−ρδr - \rho \delta from the buffer (direct extension of finding from Thm. 1 that greedy reward rate iteration can approximate ρ∗\rho^*), and PPO is a practical method for first-order approximation of natural policy gradient (extension of finding from Thm. 2 that we need natural policy gradient for de-coupling).

Weakness 2 and Question 1

Weakness 2: The proposed experimental results do not isolate the effect of the proposed policy gradient objective. Question 1: Could you isolate the effects of the modified PPO reward and that of the buffer-selection?

We answer both concerns together since the questions are asking for an additional ablation experiment. Please see our additional ablation below, where we isolated the effect of reward-rate on MLE-Bench by using the same buffer rule as vanilla RL's by selecting samples keyed by reward only. The only difference between vanilla and ablation is the reward signal, and the evaluation is completely the same.

RPG vs VanillaAblation vs Vanilla
Average improvement across 22 tasks at the final budget (3 trials)+19.2%+18.6%

Question 2

Question 2: How does RPG compare with RL using the same time-aware buffer selection and a fixed coefficient for the time penalty?

We have conducted additional experiment by using scalarized reward r−αδr - \alpha \delta with α\alpha tuned through a grid search for α∈(0.001,0.05,0.1)\alpha \in (0.001, 0.05, 0.1). Our results are shown below. It is worth noting that the grid search process is very computationally expensive, and RPG doesn't require hyperparameter search.

RPG vs VanillaRPG vs Linear Scalarization
Average improvement across 22 tasks at the final budget (3 trials)+19.2%+6.5%

Question 3

Question 3: Thm. 1 assumes a bounded approximation error for the greedy reward rate. Since the optimlal poilcies are known in the synthetic bandit experiments, can you report the estimation error of the NIW and show that the condition in Thm. 1 is satisfied?

We thank the reviewer for the advice. We provide the result below by calculating the ratio ρ~k+1/ρ(uρk)\tilde \rho_{k+1} / \rho(u_{\rho_k}). For the assumption to satisfy, we need 1/(1+γ)≤ρ~k+1/ρ(uρk)≤1+γ,1/(1+\gamma) \leq \tilde \rho_{k+1} / \rho(u_{\rho_k}) \le 1+\gamma,, and the table below shows us that this is consistently the case, where we take the median across environments and arms for each context setting. We also show visualizations of the ratio in our revised paper in Appendix D.3 and edited the main text to reflect this new information.

|X|Median of ratiosRange of ratios% steps with ratio in [0.80, 1.25]% steps with ratio in [0.5, 2]
41.0270.997 – 1.05197.899.8
81.0610.987 – 1.17494.099.7
161.0671.014 – 1.13692.699.4
321.0620.987 – 1.11394.499.6
641.0401.003 – 1.08895.899.9

Thank you for the clarifications and additional experiments. While these address some of my empirical concerns, I remain concerned about the limited applicability of the theoretical results to the proposed algorithm. I will maintain my original score.

We thank the reviewer for the subsequent response. We would like to kindly follow up for further clarification.

Our motivation for the theoretical analysis is not to justify each design choice of the proposed algorithm. Rather, it is to conduct analysis on what is unique about reward-rate RL, which is suggested by Thm.2 that even in the best case scenario (with exact expectation and zero estimation error), we would have a coupling issue if we use vanilla policy gradient and thus necessitates the use of natural policy gradient. For this reason, we believe our theoretical analysis is a meaningful contribution to performing policy-gradient-style updates in the reward-rate RL setting, and this observation holds for all reward-rate RL bandit settings. The observations from theory then shed light on designing a practical algorithm on MLE-Bench and NanoGPT.

If possible, would the reviewer kindly inform us what kind of theoretical results would be considered as a strong contribution? We look forward to your feedback! Thank you once again!

Reviewer_ds7aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to handles reinforcement learning tasks with variable time taken for each step. The problem is formulated using Semi-Markov decision process framework. The policy dependent performance metric is defined as a ratio of cumulative reward and cumulative time taken. Consequently, the optimal policy is trying to optimize for reward per unit time. Subsequently, policy gradient theorem is proposed and performance metric is estimated using estimator based on Normal-Inverse-Wishart posterior. The proposed algorithm is evaluated on MLE-Bench tasks and NanaGPT speedrun challenge.

Critical strengths
  1. The problem statement of optimizing reward per unit time is very important considering the aspect of execution time of each action.
  2. The policy gradient provided in Eq. 2 represents a non-trivial result and helps is proposing a practical algorithm sensitive to the execution time of each action.
  3. Reward rate estimator based on Normal-Inverse-Wishart posterior, alleviates the issue of estimating reward rate using on-policy rollouts, which could be a time intensive process.
  4. The proposed algorithm outperform vanilla RL on 22 MLE-Bench task w.r.t mean reward performance. Further, on NanoGPT task it obtains the target validation loss faster compared to vanilla RL.
Critical weaknesses
  1. There is not justification provided for the recursion used in Eq. 16 for differential state and action-value function. Further, Eq.16 just mentions δ\delta without specifying that whether it is expectation of δ\delta w.r.t. the stationary distribution or δ\delta for a particular state.
  2. It is difficult to evaluate the correctness of policy gradient theorem because of lack of justification for the correctness of the differential value functions.
  3. Insufficient description about greedy reward rate estimator fpf_p in the main text of the paper. The greedy reward rate estimator seems to be one of the main contributions of the work, however, it is not even mentioned where details could be located in the appendix.
  4. Expression for bt(u)b_t(u) is missing from section 2.2. Further, no reference to another work or appendix is provided to specify it.
  5. For the experiments, only vanilla RL is used as baseline. It would be better to compare the proposed algorithm with GRPO as well.
Questions for authors

What is meant by vanilla RL baseline?

3 comments

We thank the reviewer for the constructive feedback. Please see our responses to each of the raised concerns below.

Weakness 1 and 2

Weakness 1: There is not justification provided for the recursion used in Eq. 16 for differential state and action-value function. Further, Eq.16 just mentions without specifying that whether it is expectation of w.r.t. the stationary distribution or for a particular state. Weakness 2: It is difficult to evaluate the correctness of policy gradient theorem because of lack of justification for the correctness of the differential value functions.

We thank the reviewer for pointing out this issue on clarity. We didn't choose to justify Eq. 16 because it is entirely based on previous work from [1], and differential value function is a well-established, correct concept [2]. To that end, we have edited the text correspondingly to make the explanation clearer. As for the question on what the expectation is taken with respect to, by definition of conditional expectation, the state ss (or ss and aa) is fixed, so it is taken wrt transition dynamics p(r,δ,s′∣s,a)p(r,\delta,s'\mid s,a) and the policy π(a∣s)\pi(a\mid s) fixing the state (for state value function Vπ(s)V^\pi(s)) or state and action (action-value function Qπ(s,a)Q^\pi(s,a)).

Weakness 3

Weakness 3: Insufficient description about greedy reward rate estimator fρf_\rho in the main text of the paper. The greedy reward rate estimator seems to be one of the main contributions of the work, however, it is not even mentioned where details could be located in the appendix.

We thank the reviewer for the advice. As noted in Algorithm 1 using inline code comments highlighted in green, our reward rate estimator for our study is "a Normal-Inverse-Wishart (NIW) posterior, but it can also be Robbins-Monro or neural networks." The specific initialization of fρf_\rho is also described in Section 3.1 "The estimator fρf_\rho can be anything that estimates the greedy reward rate, such as Bayesian inference, Robbins-Monro iteration, or neural networks." In Appendix C.2, we also specifically described how we fit the NIW estimator empirically.

Weakness 4

Weakness 4: Expression for bt(u)b_t(u) is missing from Section 2.2. Further, no reference to another work or appendix is provided to specify it.

We thank the reviewer for the suggestion. As noted at the end of Section 2.2, "more details on the algorithm's construction of ctc_t and bt(u)b_t(u) are in Appendix A.2," and in Appendix A.2, we explicitly described bt(u)b_t(u).

Weakness 5 and Question 1

Weakness 5: For the experiments, only vanilla RL is used as baseline. It would be better to compare the proposed algorithm with GRPO as well. Question 1: What is meant by vanilla RL baseline?

By vanilla RL, we are referring to using PPO on the reward rr only. As described in Section 4.2, "Our baseline is vanilla RL, where rewards are performance scores only and time budget is \delta_\max." Nonetheless, we have edited our text further to make the point clearer.

We thank the reviewer for suggesting GRPO as a baseline. We would like to kindly note that the main contribution of RPG is an alternative view towards time-aware RL through the reward-rate formulation, which corresponds to using the relative reward r−ρδr - \rho \delta as the signal. Therefore, comparing with GRPO is not a primary focus of our work since GRPO is essentially about an alternative way to calculation advantage through group-based normalization, rather than being time-aware.

References

[1] Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. NeurIPS 1999.

[2] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Second edition, MIT Press, 2018.

Thank you for addressing my concerns. However, I would maintain my score.

We thank the reviewer for the response!

Reviewer_CvjpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Reward-rate Policy Gradient is designed for machine-learning-engineering agents whose actions take different amounts of time. It replaces raw reward with r-\rho\delta, where \delta is execution time and \rho is a reward-rate estimate based on past rewards and durations. The implementation uses a Normal-Inverse-Wishart model, a high posterior-predictive quantile, PPO, and a self-improvement buffer ranked by this adjusted reward.

The theory studies an approximate Dinkelbach-style rate iteration and an exact population, tabular natural-policy-gradient method for a finite contextual bandit with variable action durations. Experiments cover synthetic bandits, 22 MLE-bench tasks, and NanoGPT optimization with a separate frontier-model implementer. The reported gains are a 19.2% trimmed mean relative improvement across MLE tasks and an 85.7% improvement in the margin below a NanoGPT loss threshold.

Critical strengths
  1. The paper addresses an important cost that agent evaluations often overlook. Engineering actions can have very different runtimes, so an adaptive time penalty is worth studying. The treatment of invalid attempts, duration floors, past attempts, and resource limits gives a concrete account of how the system works.

  2. The idealized analysis and synthetic experiments provide useful insight. The rate-iteration and tabular natural-gradient results help explain why estimating the greedy policy's rate might be useful. The synthetic experiments use separate tuning and evaluation seeds, with 100 evaluation seeds. The real-agent experiments also cover multiple tasks. These are strengths, even though they do not establish the guarantees for the implemented system.

Critical weaknesses
  1. The experiments change several factors at once and leave final evaluation independence unclear. In NanoGPT, Eqs. (64)–(65) change the base reward, bonuses, scales, failure penalties, and buffer ranking between RPG and the baseline. The comparison therefore cannot isolate the adaptive rate term. On MLE-bench, grader scores feed into PPO and later prompts (pp. 28–30), but a separate untouched final evaluation split is not clearly described. The original benchmark's validation interface checks submission validity without returning a score, so this change matters. The reported time budget also counts executed-program time rather than the full cost of generation, training, implementation, and judging. Together, these issues weaken the claim of improved end-to-end efficiency. Original MLE-bench protocol, Section 2.3

  2. The practical rate estimator lacks the justification claimed for it. Theorems 1–2 assume controlled errors in the greedy-rate estimate or exact population/tabular updates. The paper does not show that the NIW quantile estimator and PPO meet these conditions. Appendix C.1 also claims that retaining samples when the sampled arm equals the greedy arm gives data for the greedy policy. In general, this selection reweights contexts by their probability of choosing that arm. The two-context example below has true greedy rate 0.4 but a gated ratio of 0.4857. The paper acknowledges approximate estimation, and this example does not refute the idealized theorem. It does show that the estimator's explanation needs correction and that its practical benefit needs stronger evidence.

Questions for authors
  1. Which data splits or labels supply the MLE training reward, prompt feedback, hyperparameter selection, and final Table 1 scores? Were any final scores kept unseen until training and selection were complete? If the same grading feedback is reused, please state exactly what the results measure and whether an untouched evaluation is available. Also explain how the common final budget is selected when chain durations differ. This is central to interpreting the reported benchmark gains.

  2. Do the gains remain when both methods use the same base reward, bonuses, failure penalty, buffer ranking, and compute allocation, changing only the adaptive time-price term? Please point to any existing ablation that provides this comparison. For NanoGPT, is there a tuned fixed-time-penalty baseline under the same reward definition? Without such a comparison, which part of the gain can be attributed specifically to RPG?

  3. Keeping samples only when the greedy arm is chosen reweights each context by the probability of choosing that arm. Does the implementation correct this selection bias, for example through per-context weighting or propensity weights? If not, what rate is the NIW estimator intended to estimate when the policy changes and history is selected by top-k? Please clarify whether any measured error or bound connects that estimate to the greedy rate required by the theory.

  4. Can you separate executed-program time from total generation, PPO training, implementer, and judge costs, and report the available variation across independent runs? Please also define the non-monotone “best crossing validation loss” curve in Figure 4: is it a running minimum, a current-step value, or a statistic over changing chains? These details would help determine whether the claimed benefit is faster program execution, better solutions within a particular budget, or lower end-to-end engineering cost.

2 comments

We thank the reviewer for the constructive feedback. Please see our responses below.

Weakness 1

In NanoGPT, Eqs. (64)–(65) change the base reward, bonuses, scales, failure penalties, and buffer ranking between RPG and the baseline. The comparison therefore cannot isolate the adaptive rate term.

Given that NanoGPT scores a submission by time to reach the target loss, the most intuitive baseline is to minimize time to reaching the target loss through Eq.(65). However, since time is on a completely different scale than RPG's reward signal of loss per unit of time, we must use alternative scaling and bonuses to make the baseline work. To ensure a fair comparison, we have tuned all the coefficients through grid search for the baseline. As for the ranking from the buffer, we have conducted an additional ablation on MLE-Bench by keeping everything same except for the reward-signal, and the result is shown below.

RPG vs VanillaAblation vs Vanilla
Average improvement across 22 tasks at the final budget (3 trials)+19.2%+18.6%

... a separate untouched final evaluation split is not clearly described...

We would like to kindly note that in the auto-research and MLE literature, it is generally the norm that people evaluate on a fixed task [3, 4]. This setting is different from supervised learning setting where we train on a fixed training set through empirical risk minimization and estimate the generalization error on the test set. In the auto-research setting, our expectation is taken with respect to the environment transition dynamics which is unknown. Other works also conduct evaluation on per-task basis as well [5]. Therefore, our paper follows this norm and do not use held-out tasks during evaluation.

The reported time budget also counts executed-program time rather than the full cost of generation, training, implementation, and judging.

We would like to point out that our primary motivation for this work is to perform efficient RL updates when having variable time consumption during the MLE training process, which is different from the typical LLM reasoning setting that optimizes for inference cost. This is also a completely different setting than LLM reasoning, where the variation in time comes from the external environment through training the MLE script by using CPUs/GPUs, whereas LLM's own inference cost is a function of the policy πθ\pi_\theta itself. For this reason, we believe estimating only the executed-program time is sufficient.

Weakness 2

The practical rate estimator lacks the justification claimed for it. Theorems 1–2 assume controlled errors in the greedy-rate estimate or exact population/tabular updates. The paper does not show that the NIW quantile estimator and PPO meet these conditions.

We would like to kindly highlight that our motivation for the theoretical analysis is to study the training dynamics of reward-rate RL by separating the problem of finite-sample estimation from training. In particular, we believe having an accurate finite-sample reward-rate estimate is a problem of estimating a ratio of expectation of sum of random variables over time, which is not central to studying what is unique about the reward-rate RL training dynamics.

For this reason, our analysis is focused on answering these questions under exact expected values: (Q1) under what conditions can we obtain an approximate (or exact) value of optimal ρ∗\rho* through iterating the policy, since ρ∗\rho* is central to finding the problem's solution; (Q2) even with exact estimation, what is unique about reward-rate RL's training.

To that end, Thm.1 answers (Q1) by showing that the iteration on the greedy reward rate converges near ρ∗\rho* when γ≠0\gamma \neq 0 and to exactly ρ∗\rho* when γ=0\gamma = 0. To answer (Q2), Thm.2 further analyzes the training dynamics when that estimation error is removed (γ=0\gamma = 0), which informs us that reward-rate RL has its unique coupling issue even with exact estimation, and that issue is addressed through natural policy gradient.

The posterior-based estimator and PPO is an extension of our theoretical finding into practice. The posterior estimate approximates the greedy reward-rate by fitting on greedy samples ranked by r−ρδr - \rho \delta from the buffer (direct extension of finding from Thm. 1 that greedy reward rate iteration can approximate ρ∗\rho^*), and PPO is a practical method for first-order approximation of natural policy gradient (extension of finding from Thm. 2 that we need natural policy gradient for de-coupling).

...The two-context example below has true greedy rate 0.4 but a gated ratio of 0.4857. The paper acknowledges approximate estimation, and this example does not refute the idealized theorem...

The two-context example is not shown. Would it be possible for you to clarify further on the example?

Question 1

Which data splits or labels supply the MLE training reward, prompt feedback, hyperparameter selection, and final Table 1 scores?

As discussed above, we perform auto-research style training based on a fixed task of optimizing performance for MLE-Bench, and the reward is provided by the MLE-Bench grader on a test set hidden from the agent. The agent's submitted scripts interact with MLE-Bench and creates feedback from the environment which becomes prompt feedback. Our proposed method RPG does not involve a hyperparameter to be tuned and we use the default PPO hyperparameters. Table 1's scores are based on the average performance within a fixed time budget across all 22 tasks.

Also explain how the common final budget is selected when chain durations differ.

As discussed in our paper, "The final time TT is decided by taking the cumulative time containing 95% of the self-improvement chains."

Question 2

Do the gains remain when both methods use the same base reward, bonuses, failure penalty, buffer ranking, and compute allocation, changing only the adaptive time-price term?

As previously discussed, we ran an additional ablation study using vanilla RL's buffer selection rule, thus keeping everything same except for the reward signal. Our result is below.

RPG vs VanillaAblation vs Vanilla
Average improvement across 22 tasks at the final budget (3 trials)+19.2%+18.6%

For NanoGPT, is there a tuned fixed-time-penalty baseline under the same reward definition?

We did not ran a fixed-time penalty baseline because it is computationally heavy to perform hyperparameter search on the penalty term in NanoGPT, since each grading attempt on a generated script from the LM uses 8 A100s. However, we did ran ablation on a fixed-time baseline for MLE-Bench, where the computation is less heavy, and our finding is shown below

RPG vs VanillaRPG vs Linear Scalarization
Average improvement across 22 tasks at the final budget (3 trials)+19.2%+6.5%

Question 3

Keeping samples only when the greedy arm is chosen reweights each context by the probability of choosing that arm. Does the implementation correct this selection bias, for example through per-context weighting or propensity weights?

Our theoretical analysis informs us that greedy reward rate iteration can converge near the optimal rate, thus we believe it is essential to have accurate estimate of the greedy rate, and it does not need a re-weighting to remove the bias from the greedy selection.

Question 4

Can you separate executed-program time from total generation, PPO training, implementer, and judge costs, and report the available variation across independent runs?

Similar to previous responses, we believe that the unique problem practitioners face when applying RL on MLE is the variable training time incurred through the external, unknown environment. This property separates our MLE task from the already well-studied settings such as optimizing LLMs' inference and training costs. To that end, we believe it is not essential to report the time incurred by inference and training. Moreover, inference and training costs are from the policy πθ\pi_{\theta} itself, whereas our study focuses on letting the agent to learn to act efficiently in the external environment.

Please also define the non-monotone “best crossing validation loss” curve in Figure 4: is it a running minimum, a current-step value, or a statistic over changing chains?

Each self-improvement chain runs a sequence of attempts over steps, and every attempt counts toward the chain’s time. An attempt “crosses” if its final validation loss is below the 3.28 target. For one chain, the best crossing validation loss at a budget tt is the lowest validation loss among its crossing attempts that finished within its first tt seconds of total run time. We have updated our paper to clarify this definition.

References

[3] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. ICML 2024.

[4] Hjalmar Wijk et al. RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts. ICML 2025.

[5] Zhengyao Jiang et al. AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Meta Response

We thank the reviewers to their constructive feedback. We gather some common concerns among the reviewers and present our general response below

  1. Theoretical analysis is in an ideal setting, different from empirical implemented methods: we would like to highlight that our analysis is to provide insight on the unique nature of reward-rate RL, which is revealed by Thm.1 by informing us that greedy iteration on the reward rate can approximate the optimal. Thm.2 tells us that reward-rate RL has a coupling problem even with exact expectations, and that error is removed throught natural policy gradient. In order to complete the proof, we assume exact expectation and bounded approximation error to focus on analyzing the training dynamics rather than being complicated with finite-sample estimation. These two findings shed light on our design choices for practical implementation on MLE-Bench and NanoGPT. Our intention is not do conduct theoretical analysis on the empirical experimental settings.
  2. Difference in reward shaping and buffer rules complicates the conclusion: For NanoGPT, since the task scores submissions by time to reach the target loss, we use vanilla RL optimizing for this quantity directly as the baseline. This design inherently places the baseline's score on a different scale than RPG, hence giving the difference. To ensure fair comparison, we first tune the baseline's reward weights and then compare with RPG. As for the buffer selection rule, we conducted additional ablation study where we keep everything equal except the reward signal, and our finding is similar as before, with 18.6% average improvement over vanilla RL on MLE-Bench.
  3. Comparison with a fixed-time-penalty baseline: we have added this baseline with reward as r−αδr - \alpha \delta, which is tuned through hyperparameter search with α∈(0.001,0.05,0.1)\alpha \in (0.001, 0.05, 0.1). We found that RPG obtains higher average improvement than the tuned fixed time penalty baseline by 6.5%. It is worth noting that RPG doesn't require hyperparameter tuning, and the grid search is computationally expensive.
  4. Why not include time for LLM inference and training: our motivation for this work stems from teaching agent to learn efficient interaction with an external, unknown environment (in this case, the MLE pipeline). This is a different problem than optimizing efficiency for training and inference of the LLM policy πθ\pi_\theta itself. Thus, we only measure time incurred on the environment which is the MLE training wall-clock time.

We have also updated our paper submission accordingly, our changes are in red-colored text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.